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Executive  Summary 

The  study  and  findings  described  in  this  report  are  part  of  a  larger  effort  that  is  focused  on 
Human  Systems  Integration  (HSI)  in  acquisition  and  was  conceived  and  sponsored  by  the  Air 
Force’s  711th  Human  Performance  Wing’s  Human  Performance  Optimization  Division  (711 
HPW/HPO).  Other  projects  in  this  effort  include  HSI  in  the  Systems  Engineering  Plan,  in  the 
Life  Cycle  Management  Plan,  and  in  the  Systems  Requirements  Document. 

This  study  on  HSI  in  Test  and  Evaluation  (T&E)  examined  T&E  processes,  planning  teams, 
guiding  documentation,  and  roles  to  uncover  opportunities  for  improving  the  visibility  and 
treatment  of  HSI  in  T&E.  The  study  team  engaged  Subject  Matter  Experts  (SME)  from  the  T&E 
and  HSI  communities  in  a  one  day  workshop  to  identify  focus  areas  for  the  project.  The 
participants  identified  the  following  five  areas  as  needing  attention:  1)  testable  requirements;  2) 
guidance  documents;  3)  linkages  with  existing  tests;  4)  improved  measurement  methods;  and  5) 
usability  shortfalls. 

The  study  team  conducted  focused  interviews  and  performed  document  reviews  to  shape 
investigations  for  each  of  the  five  areas.  The  investigations  generated  recommendations  for 
modifications  to  Air  Force  policy,  instructions,  and  guidance  for  T&E.  These  are  included  as 
appendices  to  this  report. 

A  key  feature  of  the  study  was  the  development  of  26  HSI-related  sample  entries  for  the  T&E 
Top-Level  Evaluation  Framework  Matrix  (the  “Framework”).  The  Framework  is  a  required 
section  of  Test  and  Evaluation  Strategy  and  Test  and  Evaluation  Master  Plan  documents.  The 
Framework  displays  the  program’s  developmental  and  operational  test  process  at  a  glance  by 
tying  together  high-level  user  requirements  (including  Key  Performance  Parameters),  key  test 
measures,  critical  test  parameters,  and  test  methodologies.  The  Framework  had  been  identified 
by  workshop  participants  as  a  primary  opportunity  for  including  HSI  in  T&E  processes,  and  thus 
will  become  a  key  enabler  for  better  integration  of  HSI  and  T&E. 

The  HSI-related  Framework  examples  developed  during  this  task  link  HSI  to  existing  tests  and 
program  decisions,  describe  measurement  methods,  and  include  testable  requirements  and 
several  usability  examples.  Combined  with  the  Air  Force  Instruction  recommendations,  these 
Framework  examples  address  all  of  the  five  focus  areas  that  had  been  recommended  by  SMEs. 
The  26  examples  are  included  as  an  appendix  to  this  report. 

The  study  team  recognized  two  key  factors  that  have  hindered  better  integration  of  HSI 
concepts  into  the  T&E  community.  First,  T&E  in  the  Air  Force  is  distributed  over  a  number  of 
organizations.  While  the  same  policy  and  guidance  governs  the  distributed  activities,  the 
variation  of  roles  and  missions  prevents  complete  standardization  of  practice  from  one  group  to 
another.  Secondly,  the  understanding  of  the  HSI  concept  is  not  strong  within  the  T&E 
Community.  Similar  to  many  other  Air  Force  communities,  many  individuals  in  the  T&E 
Community  still  equate  HSI  with  Human  Factors  Engineering  (HFE).  Quite  a  few  individuals 
encountered  were  performing  HSI-related  testing  without  recognizing  it  as  being  HSI-related. 
T&E  subject  matter  experts  requested  more  information  about  HSI  and  the  identification  of  HSI 
points  of  contact  that  they  could  draw  upon  for  their  day-to-day  needs. 

The  Framework  examples  provide  an  opportunity  to  address  both  of  these  hurdles.  As  the 
study  team  was  unable  to  discover  previously  developed  examples,  these--for  HSI--may  be  the 
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first.  By  consolidating  these  examples  into  a  single  library,  they  will  be  readily  available  for  use 
by  the  T&E  Community  for  training  purposes  and  for  insertion  into  T&E  guidance 
documentation.  This  will  also  raise  Community  awareness  of  the  contributions  of  HSI  to  mission 
capabilities,  and  will  facilitate  a  consistent  understanding  of  HSI  within  the  T&E  Community. 


Introduction  and  Background 

The  scope  of  this  project  was  to  analyze  current  Test  and  Evaluation  (T&E)  processes  and  to 
recommend  ways  and  methods  to  insert  Human  Systems  Integration  (HSI)  considerations  within 
the  T&E  Community  that  would  allow  human  aspects  related  to  system  design  to  be  tested  and 
evaluated.  The  intent  was  to  identify  and  develop  means  for  greater  HSI  integration  into  T&E 
activities  and  products  that  would  result  in  capabilities  that  are  more  operationally  suitable,  safe 
and  effective. 

T&E  covers  the  entire  spectrum  of  the  Department  of  Defense  (DoD)  acquisition  life  cycle.  This 
SURVIAC  task  did  not  address  laboratory  testing  during  technology  development,  [operational] 
capability  testing  of  already-fielded  systems,  systems  deployed  through  rapid-response  / 
prototyping  processes,  or  the  process  for  upgrades  and  modifications  (aka  1067  process). 

These  have  been  deferred  for  parallel  or  future  study. 

The  Performance  Work  Statement  (PWS)  described  the  following  areas  of  work: 

V  Analyze  Existing  Test  and  Evaluation  Process:  determine  the  extent  of  HSI 
consideration  within  the  T&E  process  and  related  requirements  documents. 

2.  Identify  HSI  Personnel  Involvement  with  T&E  Process:  identify  current  areas  of 
opportunity  where  HSI  personnel  may  become  involved  with  the  T&E  process  and 
determine  how  HSI  personnel  can  become  involved  with  the  T&E  process. 

3.  Identify  T&E  Personnel  Involvement  with  HSI  Process:  identify  current  areas  of 
opportunity  where  T&E  personnel  may  become  involved  with  the  HSI  process  and 
determine  how  T&E  personnel  can  become  involved  with  the  HSI  process. 

4.  Identify  Gaps  Where  T&E  Personnel  Can  Influence  Requirements  Documents:  identify 
areas  for  inserting  HSI  into  T&E  processes  and  requirements  documents. 

5.  Identify  Areas  for  HSI  Requirements  within  T&E  Processes  /  Documents:  identify  gaps 
where  T&E  personnel  can  assist  in  authoring  HSI-related  thresholds  and  objectives 
within  requirements  documents. 

6.  Provide  Recommendation  for  HSI  Metrics  with  T&E  Processes  /  Documents:  provide 
recommendations  for  inserting  HSI  metrics  into  T&E  processes  and  related 
documentation  in  order  to  monitor  the  human  related  aspects  for  system  development 
and  design. 

To  address  these  work  areas,  the  team  adopted  the  following  three  courses  of  action: 

SME  Interactions.  The  study  team  was  able  to  satisfy  the  first  five  work  areas  (as  listed  above) 
by  conducting  technical  interchange  meetings  and  structured  interviews  with  SMEs  from  both 
communities.  The  interview  plan  included  attending  HSI  and  T&E  conferences  at  which  large 
numbers  of  those  on  the  interview  list  were  expected  to  be  present. 

Document  Analyses  and  Recommendations.  The  study  team  was  able  to  satisfy  the  first, 
fourth,  and  fifth  work  areas  (as  listed  above)  by  reviewing  acquisition  and  T&E  policy  and 

UNCLASSIFIED 

2 

Distribution  A.  Approved  for  public  release;  distribution  unlimited.  88ABW-201 2-6455;  12  December  201 2 


UNCLASSIFIED 


guidance  documents,  and  making  recommendations  for  inserting  language  and  guidance  that 
would  influence  HSI’s  inclusion  in  T&E. 

The  study  team,  in  collaboration  with  the  government,  identified  the  following  three  T&E  policy 
and  guidance  documents  as  providing  the  greatest  opportunity  to  improve  the  inclusion  of  HSI  in 
acquisition: 

1 .  Air  Force  Instruction  (AFI)  99-103,  Capabilities  Based  Test  and  Evaluation.  This 
instruction  is  the  primary  policy  document  for  T&E  in  the  Air  Force. 

2.  Air  Force  Manual  (AFMAN)  63-119,  Certification  of  System  Readiness  for  Dedicated 
Operational  Testing.  This  manual  provides  detailed  guidance  for  ensuring  a  new  or 
modified  capability  is  ready  for  dedicated  operational  test  and  evaluation. 

3.  The  Air  Force  Test  and  Evaluation  Guidebook.  This  unofficial  document  provides 
detailed  background  on  many  facets  of  Air  Force  T&E.  Since  it  is  an  unofficial  Air  Force 
publication,  it  has  not  been  updated  since  2007. 

The  study  team  recommended  86  specific  HSI-related  modifications  to  these  documents  based 
on  extensive  review  and  consultation  with  T&E  and  HSI  SMEs.  In  addition,  the  study  team 
recommended  97  specific  HSI  “touch  points”  to  provide  guidance  when  executing  particular 
sections  of  these  documents,  based  on  extensive  review  and  consultation  with  T&E  and  HSI 
SMEs. 

HSI  Measures.  To  satisfy  the  sixth  work  area,  the  study  team  investigated  the  T&E  Top-Level 
Evaluation  Framework  Matrix  (the  “Framework”)  as  a  tool  for  documenting  HSI  measures.  The 
Framework  is  a  required  section  of  Test  and  Evaluation  Strategies  (TES)  and  Test  and 
Evaluation  Master  Plans  (TEMP).  The  study  team  was  unable  to  discover  previously  completed 
Framework  examples  because  the  policy  was  only  recently  implemented  in  late  2009.  The 
development  of  a  set  of  HSI  examples  was  therefore  considered  a  prime  opportunity  for 
interdisciplinary  communication. 

Framework  categories  comprise  all  of  the  attributes  of  a  measure  including  the  following:  1 )  the 
attribute  to  be  measured;  2)  the  method  of  measurement;  3)  the  scale  of  measurement;  and  4) 
the  rule  (threshold  or  objective)  for  assigning  value  or  desirability  of  a  given  measurement.  The 
study  team  produced  26  HSI-specific  examples  for  the  Framework  that  capture  measurable, 
objective  requirements  for  developmental  and  operational  test  and  relate  them  to  technical 
parameters  and  test  methods.  The  26  representative  measures  are  intended  for  use  by  HSI 
and  T&E  personnel  in  crafting  requirements  and  associated  test  parameters. 

It  should  also  be  noted  that  the  Framework  is  limited  to  “critical”  test  parameters  due  to  the 
higher-level  nature  of  TESs  and  TEMPs.  While  HSI  test  parameters  are  extremely  important, 
they  may  not  rise  to  the  level  of  “critical”  as  the  word  is  used  in  the  context  of  the  Framework. 
Early  tester  involvement  during  the  development  of  requirements,  as  recommended  in  AFI  99- 
103,  is  a  primary  way  to  permit  the  thinking  promulgated  by  the  Framework  to  affect  HSI  without 
necessarily  having  HSI  parameters  in  the  Framework  of  any  specific  program. 
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Methodology 

Test  and  Evaluation  Workshop 

On  January  27,  2010,  the  Air  Force  Human  Systems  Integration  Office  (AFHSIO)  hosted  a  one- 
day  workshop  that  brought  together  twenty-six  SMEs  from  the  Air  Force  HSI  and  T&E 
communities.  The  number  of  representatives  from  the  HSI  community  was  approximately  equal 
to  the  number  who  represented  T&E.  Some  participants  could  represent  both  interests  (e.g., 
human  systems  testers). 

The  objectives  of  the  workshop  were  to  bring  the  HSI  and  T&E  communities  together  to 
accomplish  the  following:  1)  achieve  a  better  understanding  of  each  other’s’  processes  and  of 
the  participants’  perspectives;  2)  identify  opportunities  and  methods  that  would  enable 
representatives  of  the  two  communities  to  work  closer  together,  to  support  one  another,  and  to 
ensure  HSI  requirements  can  be  effectively  tested;  and  3)  define  the  path  forward  for  the  HSI  in 
T&E  project  and  for  the  HSI  Assessment  Project  (being  performed  for  AFHSIO).  One  of  the 
AFHSIO  assessment  projects  is  the  development  of  templates  for  the  TES  and  TEMP,  which 
will  be  used  for  reviewing  HSI  content  in  those  two  acquisition  documents. 

The  workshop  plan  consisted  of  a  series  of  presentations,  facilitated  discussions  on  targeted 
topics,  and  the  collection  of  action  items.  The  agenda  included  presentations  by  AFHSIO 
representatives,  T&E  representatives  from  the  Policy  and  Programs  Division  of  the  Air  Force 
Test  and  Evaluation  Directorate  (AF/TEP),  the  Air  Force  Flight  Test  Center  (AFFTC),  the  Air 
Force  Operational  Test  and  Evaluation  Center  (AFOTEC),  field  test  unit  representatives,  and 
Center  Test  Authorities  (CTA). 

Workshop  Findings:  A  theme  that  was  first  raised  at  the  workshop  and  continued  throughout 
the  study  was  a  perceived  lack  of  synergy  among  components  of  the  T&E  enterprise.  It  must  be 
noted  that  due  to  the  diverse  nature  of  T&E  across  the  Acquisition  Life  Cycle  (e.g.,  munitions  v. 
space  v.  command  &  control,  or  developmental  testing  v.  follow-on  operational  testing),  it  may 
appear  there  is  a  lack  of  a  coordinated  approach  to  T&E. 

Another  concern  first  raised  at  the  workshop  was  the  consistency  of  representation  by  HSI 
domain  representatives,  such  as  Environment,  Safety,  and  Occupational  Health  (ESOH)  and 
training  personnel,  in  T&E  activities.  For  example,  if  a  training  issue  is  discovered  in 
developmental  testing,  there  is  less  confidence  that  this  issue  is  communicated  back  to  the 
program  training  representative(s). 

These  both  can  contribute  to  a  situation  in  which  HSI  does  not  appear  to  be  comprehensively 
tested.  Again,  early  tester  involvement  during  the  development  of  requirements  can  help  to 
mitigate  this.  SMEs  interviewed  during  this  study  indicated  that  human  factors  and  ESOH 
should  be  tested  as  part  of  developmental  testing.  However,  ESOH  testing  has  not  been 
institutionalized  in  the  Air  Force.  Manpower,  personnel,  training/training  support,  and  human- 
machine  interfaces  may  be  tested  as  part  of  logistics  T&E. 

HSI  Awareness:  The  aforementioned  diversity  within  the  T&E  Community  means  there  will  not 
be  a  singular  point  of  interface  for  HSI  with  T&E.  HSI  representatives  will  need  to  align  with  the 
test  organizations,  or  with  test  managers  at  program  offices  and  with  major  commands 
(MAJCOMs),  air  logistics  centers,  detachments,  product  centers,  and  field  test  units  which  also 
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have  test-coded  positions.  The  only  Air  Force  T&E  organizations  with  an  organic  HSI  presence 
(primarily  human  factors)  are  the  AFFTC  and  AFOTEC. 

The  T&E  Community  asked  for  help  in  becoming  better  informed  about  HSI  issues  so  they  could 
make  improvements  in  this  area.  They  recommended  HSI  involvement  in  Integrated  Test 
Teams  (ITTs)  and  ITT  subgroups  as  a  conduit  for  raising  HSI  issues.  This  would  allow  HSI  to 
maintain  persistent  visibility  throughout  the  life  cycle.  ITTs  should  be  formed  in  the  Materiel 
Solution  Analysis  phase,  but  this  does  not  consistently  occur.  Even  at  that  point  in  acquisition,  it 
is  considered  too  late.  The  Aeronautical  Systems  Center’s  Center  Test  Authority’s  (ASC/AQT) 
“Test  Early-Start  and  Issues”  initiative  calls  for  a  test-stakeholders  meeting  to  consistently  occur 
immediately  after  the  Materiel  Development  Decision.  ASC/AQT  representatives  suggested 
that  HSI  participate  as  an  early  stakeholder. 

There  was  an  attempt  to  find  goal  alignment  between  the  HSI  and  T&E  communities. 
Participation  by  T&E  and  HSI  representatives  in  High  Performance  Teams  (HPT)  that  develop 
operational  requirements,  and  HSI  participation  in  ITTs  are  ways  to  achieve  goal  alignment. 

HSI  in  Requirements:  Many  of  the  T&E  SMEs  that  were  interviewed  recognized  the 
importance  and  relevance  of  HSI  requirements.  The  T&E  Community  emphasizes  that  it  does 
not  write  requirements;  testers  provide  help  and  advice.  This  is  consistent  with  an  important 
aspect  of  T&E  -  impartiality.  They  can  ensure  that  the  requirements  that  are  brought  forward  by 
users  are  stated  in  testable  forms,  if  that  is  possible.  This  is  one  of  the  reasons  that  the 
philosophies  of  “early  tester  involvement”  and  “up  front  and  early”  are  strongly  promoted  by  the 
T&E  Community.  It  was  noted  that  not  all  requirements  are  testable;  for  example,  lethality  is 
measurable  but  sometimes  not  testable  in  a  cost-effective  manner,  so  modeling  and  simulation 
are  used  as  alternatives. 

HSI  in  requirements  writing  is  supported  by  HSI  representatives  at  both  the  MAJCOMs  and  at 
the  AFHSIO.  The  T&E  voice  at  the  HPT  can  help  ensure  that  human-related  requirements  are 
supported  by  appropriate  testable  or  measurable  criteria.  The  T&E  Community  knows  how  to 
test  these  HSI  relevant  requirements  and  does  so  routinely.  The  T&E  representatives  who  are 
familiar  with  these  methods  can  help  HSI  representatives  in  crafting  these  requirements.  The 
advocacy  for  the  voice  of  HSI  on  the  HPT  lies  with  the  user  and  with  the  AFHSIO. 

Operational  Concepts:  Operational  concepts  are  specifically  documents  that  provide  a 
system’s  operational  context;  i.e.,  how  a  system  will  be  used  (this  is  in  contrast  with  the  more 
formally  defined  concepts  of  operations  (CONOPS)).  They  are  one  of  the  foundations  for 
designing  operational  tests  and  are  very  helpful  for  designing  developmental  tests.  They  should 
help  to  define  test  planning  and  help  to  form  the  basis  of  operational  tests.  Customarily, 
however,  they  are  received  too  late  by  the  operational  test  team.  HSI  task  and  work  analysis 
techniques  are  suited  to  operational  concept  development.  This  is  an  area  where  HSI 
practitioners  can  help  warfighters  write  better  concepts  for  T&E. 

Testing  HSI  Requirements:  Getting  requirements  into  test  was  identified  as  another  issue.  For 
example,  ESOH  compliance  is  required  by  regulation;  ESOH  requirements  are  written,  but 
ESOH  is  rarely  tested  in  the  developmental  test  phase.  One  workshop  participant  stated  that  it 
is  considered  difficult  to  measure  “safety-ness,”  but  there  are  measurable  hazards.  Operational 
T&E  looks  at  meaningful,  mission-level  outcomes.  The  human  element  is  not  separable  in  this 
view,  but  if  there  is  a  deficiency,  testers  will  try  to  assess  what  caused  the  deficiency.  The 
difficulty  with  this  approach  is  that  finding  an  HSI-  related  requirement  that  was  missed  does  not 
equate  with  fixing  the  deficiency.  In  many  cases  budget  limitations  might  prevent  addressing 
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that  one  missed  requirement.  In  cases  such  as  this,  questions  of  verification  and  validation  and 
compliance  remain  unanswered. 

One  approach  suggested  for  institutionalizing  the  treatment  of  requirements  is  to  return  to  the 
practice  of  putting  standards  on  contract.  This  practice  would  require  contractors  to  include  the 
cost  of  doing  standards-related  work  in  their  proposals  and  make  it  more  likely  that  the  work  is 
accomplished. 

HSI  Deficiencies:  The  attendees  discussed  the  use  of  deficiency  reports  for  tracking  HSI 
issues  and  for  identifying  lessons  learned.  The  attendees  expressed  concerns  that  HSI 
deficiencies  may  be  getting  closed  without  appearing  to  be  sufficiently  addressed.  This  could 
be  the  result  of  a  lack  of  funding,  advocacy,  or  established  requirements.  While  this  may  have 
been  the  case  in  the  past,  recent  changes  to  the  technical  manual  that  governs  deficiency 
reporting  for  the  Air  Force  should  help  strengthen  this  process  and  ensure  that  deficiencies  are 
not  closed  without  being  addressed  and  reviewed  by  the  appropriate  organizations. 

SMEs  suggested  that  the  Joint  Deficiency  Reporting  System  (JDRS),  which  is  replacing  the  Air 
Force  Deficiency  Reporting  Information  System  (DRIS),  could  be  used  to  identify  common 
problems  and  to  generate  statistics  on  design  performance  with  respect  to  HSI.  For  those  few 
deficiencies  that  meet  the  necessary  criteria,  it  would  be  appropriate  to  categorize  and  track 
them  as  safety  deficiencies  via  safety  engineering  processes. 

Workforce  Development:  Initiatives,  such  as  “Test  Early-Start  and  Issues,”  demand  additional 
test-coded  positions  if  they  are  to  spread  throughout  the  Air  Force.  Building  up  the  T&E 
workforce  is  a  major  concern  of  the  Community.  New  T&E  courses  are  being  developed  and 
existing  courses  are  being  improved.  Opportunities  exist  to  insert  relevant  HSI  content  into  Air 
Force  Institute  of  Technology,  Defense  Acquisition  University  and  even  Naval  Postgraduate 
School  T&E  courses. 

Workshop  Issues:  At  the  conclusion  of  the  presentations  and  discussions,  the  important 
findings  were  collected  and  prioritized.  Additionally,  a  substantial  list  of  action  items  was 
developed  from  attendee  notes.  The  list  highlighted  questions  and  opportunities  to  be 
addressed  by  the  study  team. 

The  attendees  were  able  to  categorize  the  concerns  raised  during  the  Workshop  into  15  issues. 
The  attendees  were  asked  to  vote  on  which  of  the  issues  provided  the  greatest  opportunity  for 
improving  HSI  practice  in  T&E.  The  issues  were  then  ranked  by  number  of  votes  received.  The 
15  issues  were  as  follows: 

1)  Requirements 

2)  Guidance  Documents 

3)  Linking  HSI  to  Other  Tests 

4)  Measurement  Methods 

5)  Usability  Shortfalls 

6)  HSI  Expertise  on  Integrated  Test  Teams  (ITTs) 

7)  HSI  in  Acquisition  Contracts 

8)  Deficiency  Reports 

9)  Careers 

10)  Institutionalization 

11) Culture 

12)  Infrastructure 
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13)  HSI  Key  Performance  Parameters  (KPPs),  Critical  Operational  Issues  (COIs), 

Operational  Concepts 

14)  Enterprise-Level  Assessments 

15)  Early  Systems  Engineering  /  Rapid  Prototyping 

Workshop  “Top  Five  List:”  The  day  following  the  workshop,  the  study  team  met  with 
representatives  from  the  711  HPW/HPO  and  AFHSIO  to  debrief  workshop  findings.  The  group 
determined  that  the  study  team  should  focus  on  the  top  five  issues  that  were  identified  in  the 
workshop.  Each  of  these  five  issues  satisfies  at  least  one  of  the  six  areas  of  work  identified  in 
the  performance  work  statement: 

1)  Guide  the  development  of  testable  requirements.  HSI  relevant  requirements  are 
perceived  to  be  vague  and  subjectively  evaluated.  In  part,  this  is  due  to  training 
shortfalls  -  there  may  be  a  general  lack  of  familiarity  in  many  communities  with 
quantitative  methods  and  how  qualitative  methods  are  quantitatively  assessed.  Through 
closer  collaboration  of  T&E  and  HSI  practitioners,  HSI  requirement  statements  can  be 
made  more  complete,  clear,  and  more  fully  testable. 

2)  Insert  HSI  into  T&E  guidance  documents.  Several  documents  were  identified  by  the 
attendees,  the  primary  ones  being  AFMAN  63-119,  AFI  99-103,  and  the  AF  T&E 
Guidebook. 

3)  Link  HSI  to  other  tests.  Dedicated  HSI  testing  is  rare.  HSI  testing  is  almost  always 
embedded  in  other  types  of  suitability  testing  without  being  identified  as  a  separate  test. 

It  is  important  to  identify  opportunities  to  test  human-related  requirements  as  part  of 
customary  testing. 

4)  Improve  measurement  methods  and  incorporate  HSI.  Workshop  attendees  stated 
that  T&E  is  on  a  cusp  of  being  able  to  objectively  test  HSI-related  parameters.  Methods 
for  collecting  data,  such  as  for  ESOH  and  usability,  need  to  be  strengthened  and  the 
knowledge  of  their  use  promulgated. 

5)  Establish  approaches  that  address  usability  shortfalls.  Quantitative  usability 
measurement  methods  are  in  common  use.  Usability  requirements  can  be  easily 
derived  from  operational  suitability.  However,  requirements  that  drive  usability  testing 
are  often  absent  or  poorly  stated.  Additionally,  the  mathematical  methods  to  be  used  in 
usability  evaluations  are  not  well  understood,  especially  for  small  samples.  Workshop 
attendees  expressed  a  desire  to  improve  usability  verification  and  validation.  It  is 
unlikely  that  usability  will  rise  to  the  level  of  a  Key  Performance  Parameter  (KPP)  but 
many  considered  it  vitally  important  to  the  achievement  of  many  existing  KPPs  and  Key 
System  Attributes  (KSAs). 

During  the  debrief,  emphasis  was  placed  on  requirements  as  represented  by  the  T&E 
Evaluation  Framework  Matrix,  HSI  recommendations  for  T&E  documentation  (including 
operational  concepts),  and  how  to  link  or  embed  HSI  to  existing  tests. 

Post-Workshop  Focus  Areas:  The  study  team  took  the  “Top  Five  List”  and  developed  three 
courses  of  action  to  help  resolve  these  five  issues.  The  three  courses  of  action  were  as  follows: 

1)  Conduct  interviews  of  SMEs  in  the  HSI  and  T&E  communities. 

2)  Review  T&E  policy  and  guidance  in  order  to  provide  recommendations  for  insertion  of 
HSI  language. 

3)  Directly  apply  HSI  by  developing  HSI-relevant  examples  for  the  T&E  Framework 
(although  unstated,  the  intent  of  the  workshop  attendees  was  in  reality  to  influence  Joint 
Capabilities  Integration  and  Development  System  (JCIDS)  documents,  system 
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specifications,  contract  requirements  documents,  TEMPs,  and  test  plans,  all  of  which 
“connect”  to  the  Framework). 

The  remainder  of  this  report  and  associated  appendices  align  with  the  three  courses  of  action 
described  above. 

SME  Interactions 

Approach  and  Purpose 

After  the  January  workshop,  the  study  team  planned  to  further  explore  issues  related  to  the  Top 
Five  List  and  action  items  with  HSI  and  T&E  SMEs.  The  team  compiled  a  list  of  representatives 
from  Air  Staff,  the  operational  MAJCOMs,  Air  Force  Materiel  Command  (AFMC)  headquarters, 
AFMC  product,  air  logistics,  and  test  centers,  AFOTEC,  the  Air  Force  Research  Laboratory,  and 
the  Air  Force  Institute  of  Technology.  The  intent  was  to  find  at  least  one  HSI  SME  and  one  T&E 
SME  in  each  of  these  Air  Force  organizations  who  could  speak  authoritatively  for  that 
organization  regarding  their  observations  of  the  relationship  between  HSI  and  T&E  in  their 
organization. 

The  study  team  developed  a  detailed  list  of  questions  to  guide  interviews.  This  discussion  guide 
addressed  demographic  information  (e.g.,  the  interviewee’s  role  in  T&E,  the  interviewee’s  T&E 
responsibilities,  in  which  weapon  system  domain  the  interviewee  primarily  worked,  the 
interviewee’s  HSI  experience  /  exposure),  followed  by  a  series  of  questions  in  each  of  the 
following  areas:  Testable  Requirements  Development,  Test  Strategy,  Test  Planning,  Test 
Execution,  Test  Reporting,  and  Contractor  Documentation.  This  discussion  guide  is  included  as 
Appendix  C  of  this  report. 

Interviews  were  conducted  in  face-to-face  settings,  when  possible,  or  by  telephone,  over  a  six- 
week  period.  Face-to-face  settings  included  several  HSI  and  T&E  conferences  that  were 
scheduled  to  occur  during  the  interview  period.  The  study  team  planned  to  conduct  interviews 
in  conjunction  with  the  following  conferences: 

•  Department  of  Defense  (DoD)  Human  Factors  Engineering  (HFE)  Technical  Advisory  Group 
(TAG)  meeting,  Mesa,  AZ 

•  Air  Force  Test  Policy  Conference,  Peterson  AFB,  CO 

•  DoD  Test  Week,  Huntsville,  AL 

After  the  first  guide-driven  interview,  which  occurred  at  the  DoD  HFE  TAG,  study  team  members 
determined  that  the  interviewing  process  was  too  lengthy  to  fit  with  conference  agendas  and 
attendee  priorities  at  those  events.  The  study  team  used  these  conferences  as  opportunities  to 
identify  additional  opportunities  for  HSI  -T&E  interaction,  to  establish  contacts,  and  to  arrange 
post-conference  interview  appointments.  The  discussion  guide  was  also  streamlined  for 
subsequent  interviews. 

After  conducting  several  interviews,  respondent  answers  were  found  to  be  repetitively 
consistent  and  in  agreement  with  workshop  findings  and  with  information  collected  during  the 
Air  Force  CTA  Conference  which  preceded  the  interview  period;  these  findings  are  summarized 
below.  The  study  team  concluded  that  additional  interviews  would  have  diminishing  returns; 
therefore,  no  further  interviews  were  conducted  and  plans  to  attend  DoD  Test  Week  were 
abandoned. 
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Findings 

There  were  numerous  and  diverse  comments  presented  by  the  interview  respondents.  Many  of 
the  comments  could  be  categorized  according  to  the  Workshop’s  “Top  Five  List”.  Those 
comments  that  did  not  fit  into  these  five  categories  fell  into  a  general  category  of 
institutionalizing  HSI  within  the  T&E  culture.  The  specific  findings  are  summarized  and  grouped 
in  the  following  sections. 

Guide  the  Development  of  Testable  Requirements: 

•  Writing  HSI  Requirements:  Respondents  noted  a  void  in  the  area  of  requirements 
criteria  for  human  performance,  making  it  difficult  to  write  good  requirements.  In 
addition,  quantifiable  human  reliability  measures  are  needed.  Standards  call  for 
modeling  of  human  reliability,  but  this  is  not  done  well,  particularly  for  end-to-end 
timelines  (sensor-to-shooter).  Operators,  in  order  to  be  effective  in  stating  user-driven 
requirements,  need  to  attend  requirements  classes.  Respondents  also  noted  that  non- 
mandatory/non-required1  KPPs  are  not  frequently  seen  in  JCIDS  documents,  due  in  part 
to  the  “premium”  placed  on  KPPs  within  the  process  for  DoD  acquisition  programs. 
Finally,  the  back-end  processes  -  supply  chain,  logistics  tail,  parts  required,  ability  to 
repair  -  affecting  availability  have  much  HSI  content  and  should  be  considered  as 
valuable  source  material  when  developing  HSI  requirements. 

•  Manpower,  Personnel,  Training:  Respondents  noted  that  workload  baseline  studies 
are  usually  not  performed  in  support  of  manpower  estimates.  Also,  automation  is 
sometimes  used  to  artificially  suppress  manpower  numbers,  but  can  ultimately  result  in 
substantial  increases  in  individual  workloads. 

•  Operating  Concepts:  At  least  one  respondent  identified  an  OV-5  representation  as  the 
operating  concepts  of  greatest  tester  interest.  Operating  concepts  and  enabling 
concepts,  those  required  by  operational  testers,  are  equivalent  to  day-in-the-life  (DITL) 
scenarios.  DITL  scenarios  are  useful  because  they  help  to  execute  “Test  Like  You  Fly.” 
They  tell  testers  what  people  need  to  do  on  a  daily  basis  to  make  the  system  work. 

o  DITL  scenarios  are  also  used  to  develop  evaluation  questionnaires  for  HFE  and 
logistics  testing. 

Human-in-the  Loop  testing  is  used.  Design  engineers  are  replaced  with  a  representative 
from  a  pool  of  actual  users.  This  requires  procedures  that  cover  task  performance  time, 
errors,  etc  -  non-functional  requirements. 

•  Outcomes:  It  is  possible  to  do  a  good  job  with  requirements  and  a  bad  job  with 
implementing  them;  the  Space-Based  Infrared  System’s  ground  segment  was  given  as 
an  example.  Respondents  also  noted  that  a  crew  system  can  be  barely  effective  but  not 
suitable. 

Insert  HSI  in  T&E  Guidance  Documents: 

•  Requirements  Documents:  Testers  verify  and  validate  the  requirements  found  in  the 
JCIDS  capability  documents.  They  particularly  need  a  solid  Capability  Development 
Document. 

•  Air  Force  Policy  Documents:  Some  HSI  guidance  should  be  included  in  AFI  99-103, 
AFPAM  63-119,  and  AFI  63-101  to  trigger  T&E  and  acquisition  practitioners  to  review 
and  enhance  HSI  content  in  acquisition  programs. 


1  While  all  KPPs  must  be  met,  the  JCIDS  manual  specifically  refers  to  "mandatory"  and  "required"  KPPs;  the  use  of 
"non-mandatory"  and  "non-required"  is  merely  to  distinguish  the  "mandatory"  and  "required"  KPPs  from  KPPs 
that  are  developed  for  a  specific  capability. 
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•  Contract  Documentation:  Contract  language  should  state  that  contractors  are  required 
to  test  derived  requirements.  One  respondent  noted  that  compliance  standards  have 
been  used  very  effectively  by  the  Space  and  Missile  Systems  Center  (SMC)  to  ensure 
contractors  consider  and  include  human-related  issues  and  requirements.  Another  very 
useful  guide  is  Incorporating  Test  and  Evaluation  into  Department  of  Defense  Acquisition 
Contracts,  published  by  the  Office  of  the  Secretary  of  Defense. 

•  Test  Planning:  It  was  noted  that  test  planning  by  government  organizations  is  only  one 
piece  of  the  total  test  planning  process.  Test  planning  and  test  requirements  must  also 
be  captured  in  the  appropriate  contract  documentation  to  ensure  development 
contractors  are  adequately  planning  for  contractor-led  tests.  HSI  is  reviewing  Requests 
for  Proposals,  including  HSI  in  data  information  documents  and  placing  HSI  in  the 
Systems  Requirements  Document  (SRD).  Ensuring  the  participating  of  contractor  test 
personnel  on  ITTs  will  further  support  the  goal  of  thorough  test  planning  by  development 
contractors. 

•  Test  Reports:  There  should  be  a  separate  section  for  HSI-related  results  in  test  reports. 
HSI  is  often  piggy-backed  onto  software  testing,  and  as  a  result,  HSI  issues  will  end  up 
being  included  in  the  software  test  reports.  The  Army’s  HSI  Office  (aka  MANPRINT) 
instituted  a  separate  safety  section.  There  is  a  separate  section  for  human  factors  in 
logistics  test  and  evaluation  reporting. 

Link  HSI  to  Other  Tests: 

•  Test  Planning  Methods:  The  C-130J  program  developed  an  extremely  comprehensive 
(referred  to  as  “giant”  by  one  respondent)  test  matrix  to  determine  what  was  tested  in 
which  tests.  This  helped  to  ensure  HSI  testing  was  conducted  at  every  possible 
opportunity. 

SMC  uses  a  Critical  Report  Matrix  (CRM)  which  is  similar  to  a  Deficiency  Report  (DR), 
but  is  generated  at  the  front  end  of  the  program  life  cycle.  The  CRM  is  used  to 
coordinate  testing  and  to  ensure  critical  issues  are  tested. 

•  HSI  Test  Time:  Gaining  access  to  the  system  for  testing  can  be  an  issue  for  HSI,  but 
once  again,  early  tester  involvement  to  make  him/her  aware  of  HSI  testing  needs  is  key. 
Ensuring  that  HSI  requirements  are  properly  flowed  down  into  lower  level  specifications 
(which  should  lead  to  corresponding  lower-level  test  plans  and  procedures)  is  also 
important. 

•  Specialized  Testing:  It  is  difficult  to  piggy-back  noise  testing  onto  other  tests;  it  cannot 
be  done  when  the  system  is  being  used  (tested)  for  other  purposes. 

Improve  Measurement  Methods  and  Incorporate  HSI: 

•  Contract  Documentation:  Government  data  requirements  are  not  well  specified  in 
contract  documentation.  HSI  personnel  have  trouble  getting  access  to  raw  contractor 
data. 

•  Methodologies:  Study  methods,  such  as  the  Bedford  or  Modified  Cooper,  that  were 
used  for  workload  in  the  past,  are  not  being  used  today.  Interviewees  also  clarified 
qualitative  vs.  quantitative  measures.  People  tend  to  equate  subjective  with  qualitative 
measures  and  objective  with  quantitative.  However,  subjective  measures  can  be 
quantified  using  human  factors  techniques.  Larger  sample  sizes  also  help  with 
quantifying  subjective  data.  Following  up  questionnaires  with  interviews  is  a  method  of 
using  quantified  subjective  data  to  identify  actionable  root  causes.  Finally,  statistical 
analysis,  to  include  use  of  scientifically  based  test  design  techniques  such  as  Design  of 
Experiments,  is  important  to  validate  results. 
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•  Developmental  Test:  Traditionally,  developmental  testing  is  rarely  performed  with  real 
operators,  particularly  when  fundamental  system  principles  are  being  tested  early  in  a 
test  program.  Integrated  testing  can  provide  an  avenue  to  obtain  real  operators  for  more 
mature  developmental  test  activities  and  to  bring  in  experienced  system  operators  from 
operational  units  for  what  may  normally  be  characterized  as  developmental  test  events. 

It  can  also  embed  some  variation  from  test  to  test  when  using  multiple  operators  who 
each  have  particular  preferences. 

•  Operational  Test:  There  was  disagreement  about  methods  that  could  be  used  during 
operational  test.  At  the  workshop,  it  was  asserted  that  operators  (pilots)  could  not  be 
instrumented,  but  interviews  provided  examples  to  the  contrary.  In  one  example,  privacy 
and  data  security  of  pilot  instrumentation  results  were  key  to  operator  acceptance  of  the 
instrumentation.  It  is  not  enough  to  have  only  the  primary  system  operator  representing 
the  operational  user  when  testing.  For  adequate  testing,  other  crew  positions  should  be 
included,  as  well  as  maintainers  and  other  support  personnel.  Training  is  required  on 
experimental  psychology  techniques  that  support  shadowing  operators  during 
operational  test. 

Establish  Approaches  That  Address  Usability  Shortfalls: 

•  Suitability:  Operational  suitability  is  evaluated  in  all  operational  testing.  Some 
respondents  said  all  testing  was  testing  of  suitability.  Others  said  suitability  was  really 
tough  to  test.  An  October  29,  2009  meeting  of  senior  acquisition  personnel  asserted  that 
operational  suitability  is  their  biggest  problem,  but  there  is  no  Suitability  or  Usability  KPP. 
Respondents  said  there  would  be  except  that  suitability  is  difficult  to  measure 
quantitatively. 

•  Ease  of  Test:  Some  interviewees  asserted  that  usability  was  routinely  tested.  Others 
said  requirements  were  never  written  to  drive  usability  testing.  Some  said  usability  was 
one  of  the  toughest  things  to  test.  The  study  team  concluded  that  usability  was  poorly 
understood.  Usability  of  knobs  and  dials  is  hard  for  software  and  computer  test  people 
to  address. 

Institutionalizing  HSI:  Findings  that  did  not  clearly  fit  any  of  the  five  major  categories 

previously  identified  are  presented  in  a  broader  category  of  institutionalizing  HSI  within  the 

T&E  culture. 

•  HSI  Awareness:  Interviewees  (both  T&E  and  HSI  SMEs)  were  generally  unaware  of  the 
alignment  between  Air  Force  HSI  organizations  (the  711th  HPW  and  AFHSIO)  and  their 
own  organizations.  For  example,  there  were  product  center  organizations  that  did  not 
understand  how  the  71 1th  was  organized.  Those  interviewed  expressed  a  desire  to 
better  understand  what  comprised  HSI,  what  Air  Force  HSI  resources  were  available, 
and  how  their  organizations  could  better  align  with  those  resources. 

•  Breadth  of  HSI  Domains:  Organizations  tended  to  address  some  of  the  HSI  domains 
and  not  others.  HFE  was  routinely  addressed;  the  HFE  person  was  customarily  the  one 
that  integrated  the  other  HSI  domains.  Treatment  of  Training,  Personnel,  Manpower  and 
ESOH  was  organization-dependent.  Few  respondents  mentioned  Survivability  and 
Habitability. 

•  HSI  Presence:  Within  AFRL,  HSI  exists  in  the  71 1th  HPW;  there  is  no  HSI  presence  in 
the  other  directorates.  Sometimes  it  does  not  take  much  of  a  review  to  determine 
whether  there  are  human  issues,  but  the  review  needs  to  be  conducted.  It  was  asserted 
that  this  should  be  a  71 1th  HPW  function.  Personal  networking  plays  a  strong  role  in  HSI 
effectiveness.  One  respondent  leveraged  contacts  in  the  two-letter  organizations. 
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Standards  tended  to  be  associated  with  government  two-letter  organizations  and  each 
standard  has  a  subject  matter  expert  as  a  point  of  contact. 

•  HSI  in  Deficiency  Reporting:  There  should  be  an  HSI  representative  on  Deficiency 
Review  Boards  (DRBs).  Interviewees  reported  giving  post-test  briefs  to  DRBs;  HSI 
inputs  were  ignored.  It  is  possible  DRB  participants  are  not  familiar  with  the 
performance  and  cost  implications  and  so  are  not  as  comfortable  with  assessing  the 
consequences  of  a  human-related  deficiency  as  they  are  a  technology  deficiency. 

•  Training  Requirements:  Maintainers  should  receive  training  in  engineering  terminology 
and  processes  so  they  are  able  to  engage  with  engineers  on  common  ground.  A  real 
training  plan  specific  to  HSI  testing  also  needs  to  be  developed.  HSI  testing  is  intangible 
and  hard  to  do.  The  content  should  include  small  sample  statistical  methods,  ESOH 
testing,  questionnaire  development,  how  to  ask  questions  at  test  event  debriefs, 
documenting  the  value  of  HSI  and  logistics  testing,  and  how  to  “market”  HSI  and 
logistics  testing  to  government  and  contractor  program  offices. 

These  bulleted  findings  were  used  to  guide  recommendations  for  including  HSI  in  T&E  policy 
and  guidance  documents  and  for  selecting  and  populating  examples  for  the  Framework. 

HSI  in  Test  and  Evaluation  Documentation 

During  the  initial  workshop,  it  was  determined  that  a  highly  effective  way  to  include  HSI 
guidance  into  existing  T&E  processes  would  be  to  review  T&E  policy  and  provide 
recommendations  for  HSI-related  guidance  in  those  documents.  The  workshop  attendees 
identified  this  as  the  second  highest  priority  issue  after  the  need  to  guide  the  development  of 
testable  HSI  requirements.  Recommendations  developed  under  this  study  would  be  submitted 
for  the  next  update  cycle  of  each  of  the  impacted  T&E  documents. 

At  the  recommendation  of  several  T&E  SMEs  at  the  initial  workshop  and  in  other  venues,  the 
study  team  reviewed  three  particular  T&E  policy  documents  in  which  the  inclusion  of  HSI  would 
potentially  have  the  greatest  impact  with  regard  to  T&E  processes. 

Air  Force  Instruction  (AFI)  99-103,  Capabilities  Based  Test  and  Evaluation 

This  instruction  is  the  primary  policy  document  for  T&E  in  the  Air  Force.  It  encompasses  the 
entire  T&E  process  across  the  acquisition  life  cycle.  Figure  1  shows  how  T&E  should  start  with 
early  tester  involvement  even  prior  to  the  Materiel  Development  Decision  that  officially  begins 
the  acquisition  life  cycle.  Testers  are  expected  to  be  members  of  HPTs  that  meet  to  develop 
capabilities  documents  under  the  JCIDS  process.  These  capabilities  documents,  the  Initial 
Capabilities  Document  (ICD),  the  Capabilities  Development  Document  (CDD),  and  the 
Capabilities  Production  Document  (CPD)  are  used  to  guide  the  development  of  the  technical 
requirements  in  the  acquisition  cycle.  Early  tester  influence  in  developing  these  JCIDS 
documents  helps  ensure  requirements  are  clear  and  testable,  and  can  be  effectively  tested  and 
evaluated  throughout  the  acquisition  process. 
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Capabilities  Based  Test  and  Evaluation  (AFI  99-103) 

Figure  1.  Integration  of  the  Requirements,  Acquisition,  and  T&E  Processes 


The  study  team  recommended  24  specific  insertions  into  the  instruction  that  would  strengthen 
HSI  influence.  Of  particular  note,  the  Air  Force  HSI  Handbook  is  a  recommended  addition  to 
the  list  of  references,  and  the  definition  of  HSI  is  a  recommended  addition  to  the  “Terms.”  In 
addition,  the  study  team  provided  40  specific  HSI  “touch  points”  to  provide  HSI  guidance  when 
executing  particular  sections  of  the  instruction.  The  complete  list  of  recommended  changes  to 
AFI  99-103  is  included  in  Appendix  D  of  this  report. 

Air  Force  Manual  (AFMAN)  63-119,  Certification  of  System  Readiness  for  Dedicated 
Operational  Testing 

This  manual  provides  detailed  guidance  for  ensuring  a  new  or  modified  capability  is  ready  for 
dedicated  operational  test  and  evaluation.  AFMAN  63-119  contains  a  group  of  32  different 
templates  that  address  everything  that  is  required  in  order  to  properly  begin  IOT&E.  The 
templates  are  reviewed  by  a  working  group  comprising  personnel  from  the  program  office,  the 
Operational  Test  Agency  (OTA),  the  lead  and  supporting  operational  major  commands,  the 
prime  contractor,  and  T&E  oversight  agencies.  The  series  of  template  reviews  culminates  in  a 
presentation  to  the  Program  Executive  Officer  (PEO)  for  him/her  to  accept  the  working  group’s 
recommendation  that  the  program  is  ready  to  start  IOT&E. 


A  Microsoft  Excel  workbook  tool  has  been  developed  that  permits  the  certification  working  group 
to  assign  a  color  code  (red,  yellow,  or  green)  to  each  item  in  each  template  to  graphically 
present  the  status  of  each  template.  This  tool  also  recommends  the  responsible  parties  for 
each  item.  There  is  also  a  block  for  additional  comments  for  each  item. 


The  study  team  was  able  to  make  50  specific  recommendations  for  additional  language  in  all 
but  four  of  the  32  templates.  The  study  team  also  recommended  adding  a  specific  HSI  template 
to  address  those  HSI-related  issues  that  would  not  neatly  fall  into  any  of  the  other  templates.  In 
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addition,  the  study  team  provided  52  specific  HSI  “touch  points”  to  provide  HSI  guidance  when 
executing  particular  sections  of  the  manual  and  templates. 

Because  AFMAN  63-119  focuses  on  a  program’s  readiness  to  proceed  into  dedicated 
operational  testing,  there  is  a  risk  in  relying  on  it  as  an  all-inclusive  assessment  for  a  program’s 
entire  life  cycle.  For  example,  the  training  template  addresses  the  training  of  the  test  team  only. 
It  does  not  provide  an  assessment  of  the  complete  training  program  for  the  capability.  For 
instance,  there  may  be  cases  in  which  the  test  team  receives  only  that  portion  of  the  full  training 
program  which  is  required  to  successfully  complete  IOT&E.  For  these  cases,  the  study  team 
included  a  template  item  for  the  certification  working  group  to  clearly  identify  any  gaps  between 
the  training  requirements  being  assessed  and  the  full  requirements. 

The  complete  list  of  recommended  changes  to  AFMAN  63-119  is  included  in  Appendix  E  of  this 
report. 

The  Air  Force  Test  and  Evaluation  Guidebook 

This  unofficial  document  provides  detailed  background  information  on  the  many  facets  of  Air 
Force  T&E.  It  is  produced  by  AF/TEP.  Detailed  information  has  been  segregated  in  the  T&E 
Guidebook  in  order  to  keep  T&E  policy  documents  from  becoming  cumbersome  and  bogged 
down  with  information  that  is  not  directly  relevant  to  the  directives  being  presented.  This 
detailed  information  is  customarily  referenced  in  T&E  policy  documents.  The  T&E  Guidebook  is 
an  unofficial  Air  Force  publication.  It  has  not  been  updated  since  2007. 

The  study  team  recommended  12  specific  insertions  into  the  Guidebook  that  would  strengthen 
HSI  influence.  In  addition,  the  study  team  provided  5  specific  HSI  “touch  points”  to  provide  HSI 
guidance  when  executing  particular  sections  of  the  Guidebook  to  provide  HSI  guidance  for 
reviewing  acquisition  programs.  A  complete  list  of  recommended  changes  to  the  T&E 
Guidebook  is  included  in  Appendix  F  of  this  report. 

HSI  Touch  Points  in  Other  T&E  Documents 

The  three  T&E  policy  and  guidance  documents  that  the  study  team  has  impacted  refer  to  other 
T&E  policy  and  guidance  documents.  There  are  two  particular  references  that  bear  additional 
consideration. 

•  The  AF  T&E  Guidebook  refers  to  the  T&E  Oversight  List.  Specifically,  section  3.2.2 
includes  criteria  for  determining  whether  or  not  a  program  may  be  on  the  T&E  Oversight 
List.  The  study  team  recommends  adding  a  criterion  for  an  “extremely  high  level  of 
human-system  interaction.”  Defining  what  is  “extremely  high”  may  be  difficult  to 
establish.  In  addition,  concurrence  from  the  Office  of  the  Secretary  of  Defense  would  be 
required  to  add  this  criterion. 

HSI  T&E  Touch  Points  in  non-T&E  Documents 

The  three  T&E  policy  and  guidance  documents  refer  to  several  other  policy  and  guidance 
documents  that  are  not  exclusively  part  of  the  T&E  policy  and  guidance  series. 

•  The  integrated  requirements,  acquisition,  and  T&E  process  chart  in  Figure  1  includes 
events  prior  to  the  Materiel  Development  Decision  (not  shown  in  this  version  but  prior  to 
the  Concept  Refinement  Phase  in  this  figure).  In  order  to  fully  impact  what  is  sometimes 
referred  to  as  “left  of  A”,  changes  would  need  to  be  made  to  AFI  10-601 ,  Operational 
Capability  Requirements  Development,  and  to  the  JCIDS  process  itself. 
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•  Section  2.5  of  AFI  99-103  refers  to  operational  concepts.  In  the  past  eight  to  ten  years, 
the  definitions  of  operating  concepts,  operational  concepts,  concepts  of  operation,  etc., 
have  become  clouded.  While  AFPD  10-28  (Air  Force  Concept  Development)  and  AFI 
10-2801  ( Air  Force  Concept  of  Operations  Development)  have  provided  clarity  for  some 
of  these  definitions,  some  of  the  legacy  definitions  of  an  operating  concept  or  concept  of 
operations  have  continued  to  be  referred  to  in  the  T&E  processes.  HSI  mission  task 
analysis  techniques  can  contribute  to  generating  operational  concepts  that  supply  the 
understanding,  required  for  operational  test,  of  how  a  capability  is  expected  to  be  used 
and  sustained. 

•  As  previously  mentioned  in  the  introduction,  this  study  was  not  able  to  address  every 
area  related  to  T&E.  The  following  areas  warrant  additional  attention  because  of  their 
general  importance  to  the  T&E  Community.  The  T&E  Community  collaborates  with  and 
influences  these  areas,  but  they  are  not  under  the  T&E  Community’s  control.  These 
seven  specific  T&E  related  topic  areas  warrant  additional  attention: 

o  AFI  99-103  refers  to  AFI  16-1002,  Modeling  and  Simulation  (M&S)  Support  to 
Acquisition.  Because  of  the  close  relationship  between  T&E  and  M&S,  the  study 
team  recommends  an  HSI  review  of  this  AFI  and  the  parent  DoD  Directive 
5000.59. 

o  AFI  99-103  refers  to  AFI  63-104,  The  SEEK  EAGLE  Program.  This  program 
certifies  the  carriage  of  all  non-nuclear  internal  and  external  stores  on  Air  Force 
systems.  Several  HSI  domains  would  be  touched  upon  in  the  course  of  stores 
certification,  recertification,  and  decertification.  The  study  team  recommends  a 
more  thorough  HSI  review  of  the  SEEK  EAGLE  Program, 
o  AFI  99-103  and  the  AF  T&E  Guidebook  refer  to  Technical  Order  (T.  O.)  00-35D- 
54,  USAF  Deficiency  Reporting,  Investigation,  and  Resolution.  The  deficiency 
reporting  (DR)  process  was  originally  established  to  identify  production  and 
manufacturing  defects  in  the  field.  It  was  not  originally  developed  to  capture 
deficiencies  during  early  T&E  stages,  and  it  was  especially  not  designed  for 
software-intensive  or  HSI-related  deficiencies.  The  study  team  recommends  a 
thorough  HSI  review  of  the  AF  DR  process,  to  include  T.  O.  00-35D-54,  AFI  63- 
501  (Air  Force  Acquisition  Quality  Program),  and  T.  O.  00-5-1  (Air  Force 
Technical  Order  Program). 

o  It  is  suggested  in  AFMAN  63-119  to  use  its  templates  in  conjunction  with  DoD 
4245. 7M,  Transition  from  Development  to  Production ,  aka  “the  Willoughby 
Templates”.  The  study  team  recommends  an  HSI  review  of  DoD  4245. 7M. 
o  There  is  a  very  close  relationship  between  the  Operational  Safety,  Suitability,  and 
Effectiveness  (OSS&E)  process  as  defined  by  AFI  63-1201,  Life  Cycle  Systems 
Engineering,  and  the  T&E  process,  particularly  during  developmental  test.  The 
study  team  recommends  an  HSI  review  of  AFI  63-1 201  and  the  OSS&E  process, 
o  The  AF  T&E  Guidebook  refers  to  AFI  63-114,  Rapid  Response  Process,  and 
rapid  acquisition  process  in  general.  The  study  recommends  an  HSI  review  of 
AFI  63-114  and  the  rapid  acquisition  process, 
o  AFI  99-103  refers  to  AFI  63-103,  Nuclear  Weapons  Program  Management,  and 
AFI  63-125,  Nuclear  Certification  Program.  The  study  team  recommends  an  HSI 
review  of  both  of  these  AFIs. 

TES/TEMP/Defense  Acquisition  Guidebook  (DAG)  Chapter  9 

The  TES  and  TEMP  have  prominent  roles  in  the  T&E  process.  A  related  effort  for  the  AFHSIO 
to  provide  recommended  HSI  guidance  for  developing  a  TES  or  TEMP  includes  detailed 
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comments.  Making  specific  comments  against  the  existing  outlines  for  a  TES  or  TEMP  will 
require  recommending  updates  to  Chapter  9  of  the  DAG. 

JCIDS  Documentation 

The  study  team  initially  considered  reviewing  samples  of  actual  JCIDS  capability  documents  to 
determine  how  well  HSI-related  language  was  being  incorporated  in  capability  statements.  The 
study  team  concluded  that  examining  this  in  the  context  of  the  third  course  of  action  (T&E 
Framework)  would  provide  a  larger  return  on  investment. 

The  Test  and  Evaluation  Framework 

At  the  January  2010  Workshop,  Mr.  Chuck  Triska  (AF/TEP)  unveiled  the  Top-Level  [T&E] 
Evaluation  Framework  Matrix  (“Framework”)  to  the  HSI  community  using  the  notional  example 
shown  in  Figure  2. 
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Figure  2:  Top-Level  T&E  Evaluation  Framework  Matrix  -  notional  example. 


The  Framework  has  been  a  required  section  of  TESs  and  TEMPs  since  2009.  The  Framework’s 
purpose  is  to  show  the  correlation  between  the  KPPs/KSAs,  Critical  Test  Parameters  (CTPs), 
key  test  measures  (i.e.,  Measures  of  Effectiveness  (MOEs)  and  Measures  of  Suitability  (MOSs), 
planned  test  methods,  and  key  test  resources,  facility  or  infrastructure  needs.  When  structured 
this  way,  the  Framework  should  describe  the  most  important  relationships  between  the  types  of 
testing  that  will  be  conducted  to  evaluate  the  JCIDS  requirements  and  all  other  T&E  methods. 
Only  the  top-level  measures  are  listed. 

At  the  Workshop,  the  Framework  was  identified  as  a  primary  opportunity  for  including  HSI  in 
T&E  processes.  It  simultaneously  addresses  all  of  the  Top  Five  issues  listed  in  this  report  as  it 
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captures  requirements,  is  required  for  the  TES  and  TEMP,  links  HSI  to  existing  tests,  and 
contains  measurement  methods  that  could  include  usability.  Attendees  at  the  Workshop  debrief 
made  the  Framework  an  area  of  focus  for  subsequent  study  work. 

The  study  team’s  goal  was  to  first  use  relevant  KPPs  called  out  in  the  Manual  for  the  Operations 
of  the  Joint  Capabilities  Integration  and  Development  System  to  create  HSI-related  examples. 
Additionally,  the  study  team  set,  as  a  goal,  the  development  of  at  least  one  example  for  each  of 
the  nine  HSI  domains,  plus  at  least  one  for  the  HSI  integration  function.  Table  1  lists  the  KPPs 
that  were  used  for  example  creation,  the  degree  to  which  the  KPP  is  required  (e.g.  mandatory, 
required  or  selectively  applied),  and  the  relevance  of  the  KPP  to  HSI. 


Table  1: 

<PPs  used  to  Create  HSI-related  Framework  Examples 

JCIDS  KPP 

KPP  Type 

HSI  Relevance 

Survivability 

(of  a  manned  system) 

Mandatory 

Survivability  Domain:  The  characteristics  of  a  system 
that  reduce  detection,  and  the  probability  of  being 
attacked. 

Force  Protection 

Mandatory 

Survivability  Domain:  The  characteristics  of  a  system 
that  reduce  risk  of  fratricide,  and  that  enable  the  crew  to 
withstand  man-made  or  natural  hostile  environments 
without  aborting  the  mission  or  suffering  acute  and/or 
chronic  illness  disability,  or  death. 

Sustainment  - 
Includes: 

Availability  KPP 

Required 

Availability  of  a  materiel  system  relies  not  only  on 
hardware  and  software  readiness,  but  also  on  human 
readiness.  Thus  Availability  relates  to  manpower, 
personnel  and  training  as  well  as  to  occupational  health 
and  safety. 

Reliability  KSA 

Required 

Mean  Time  To  Repair  is  a  reliability  measure  that  relates 
to  designs  that  facilitate  rapid  access,  understandability, 
parts  manipulation  and  availability.  These  are  enabled 
by  good  human  factors  engineering  practice. 

Ownership  Cost 

KSA 

Required 

Manpower,  personnel  and  training  customarily  represent 
the  largest  life-cycle  expenditures.  Additionally,  medical 
care  and  personnel  replacement  costs  associated  with 
occupational  health  and  accidents  can  represent  a  life¬ 
long  Air  Force  investment.  Finally,  survivability  and 
habitability  failures  result  in  workforce  attrition  from 
which  recruitment  and  training  costs  will  result. 

System  Training 

Selectively 

Applied 

Often  the  costs  of  design  deficiencies  are  pushed  out  to 
the  deployment  and  operations  segment  in  the  form  of 
additional  system-specific  training.  Training,  as  an  HSI 
domain,  must  be  integrated  into  the  life-cycle 
management  plan.  For  the  ever  more  prevalent 
embedded  training,  the  training  system  is  an  integral  part 
of  the  core  capability.  HSI  is  required  to  integrate 
training  expertise  and  balance  training  needs  along  with 
other  mission  requirements. 
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While  the  study  demonstrated  the  applicability  of  the  mandatory  Net  Ready  KPP  and  selectively 
applied  Energy  Efficiency  KPP,  this  exercise  was  completed  once  the  examples  were  already 
being  vetted  by  T&E  and  HSI  SMEs.  As  a  result,  these  two  KPPs  were  not  included  for  the 
examples  created. 

The  study  team  produced  26  HSI-specific  examples  for  the  Framework  that  capture 
measurable,  objective  requirements  for  developmental  and  operational  testing  and  relate  them 
to  technical  parameters  and  test  methods.  Examples  are  contained  in  Appendix  G  of  this 
report.  The  26  representative  measures  are  intended  for  use  by  HSI  and  T&E  personnel  for 
recording  associated  top-level  test  measures,  parameters,  and  resources. 

The  examples  were  vetted  with  T&E  and  HSI  community  representatives.  The  T&E  Community 
looked  at  the  categorization  of  KPPs  and  KSAs,  and  also  looked  at  the  phrasing  of  COIs, 

MOEs,  MOSs,  and  CTPs.  These  were  judged  as  correct  and  representative.  The  HSI 
community  reviewed  the  threshold  values  and  the  test  methods  for  correctness.  Comments 
from  both  communities  were  incorporated  into  the  examples. 

Concerns  about  the  Examples 

The  26  examples  are  considered  realistic  and  representative  but  not  comprehensive.  They  do 
not  fit  all  HSI  contexts  and  cannot  be  generalized  to  all  systems  or  even  system  types.  There 
may  be  key  MOEs  and  MOSs  that  are  not  included  in  the  examples. 

A  concern,  shared  by  the  project’s  government  sponsor,  was  that  programs  would 
inappropriately  cut  and  paste  the  framework  examples  listed  in  Appendix  G,  without  the 
necessary  tailoring,  into  program  documentation  and  thus  skew  development  and  test  outcomes 
such  that  weaker,  not  stronger,  capabilities  resulted.  The  examples  are  intended  to  help  people 
from  the  T&E  and  HSI  communities  think  through  the  requirements  and  T&E  documentation 
process.  The  examples  should  be  tailored  by  a  program  to  fit  their  needs;  they  should  not  be 
adopted  without  analysis  and  consideration  of  context. 

In  response  to  this  concern,  a  tentative  decision  was  made  to  subsequently  craft  a  more 
comprehensive  library  of  Framework  examples  primarily  for  use  by  HSI  practitioners.  Because 
this  follow-on  work  is  anticipated,  the  next  section  describes  the  process  by  which  the  study 
team  created  the  examples. 

Finally,  several  T&E  SMEs  indicated  HSI  considerations  should  be  captured  in  COIs,  MOEs,  or 
MOSs.  However,  there  was  concern  about  the  number  of  COIs  that  will  be  permitted  by  the 
user  commanders  who  define  requirements.  At  the  workshop,  it  was  asserted  that  some 
commanders  are  limiting  the  number  of  COIs  associated  with  a  KPP,  and  the  rule  of  thumb  is  to 
keep  the  number  of  COIs  to  a  minimum.  However,  HSI  issues  would  more  likely  be  addressed 
below  the  COI  level  (probably  as  MOEs  or  MOSs). 

Additionally,  in  discussions  with  logistics  test  representatives  at  Edwards  AFB,  occasions  were 
noted  when  non-mandatory  KPPs  for  Decontamination,  Operability  and  RAM  (reliability, 
availability  and  maintainability)  were  used.  For  the  RAM  KPP,  supportability  and  deployability 
were  COIs.  Thus,  it  can  be  seen  that  the  same  term  can  serve  several  purposes  in  the 
framework. 
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The  Process  Used  to  Create  the  Examples 

In  order  to  create  examples,  a  representative  system  needed  to  be  used  to  help  illustrate 
examples.  Many  times,  the  systems  used  were  those  that  were  recently  and  currently  being 
developed  by  the  Air  Force.  Doing  so  made  it  easier  for  the  study  team  to  devise  COIs 
associated  with  the  system.  For  example,  the  B-2,  two-pilot  manning  example  was  used 
because  it  was  familiar  to  study  team  members.  Unmanned  Aircraft  Systems  were  selected 
because  the  issues  were  familiar,  and  the  COIs  were  relevant. 

Identifying  the  system  and  establishing  the  COIs  were  the  most  critical  parts  of  the  process. 
Once  completed,  the  COIs,  MOEs,  and  MOSs  were  stated  in  question  form  so  that  testers  could 
answer  the  questions  affirmatively  or  negatively  at  the  conclusion  of  developmental  or 
operational  testing. 

The  MOEs  and  MOSs  were  explored  to  develop  the  more  detailed  CTPs.  While  some  HSI- 
related  parameters  are  qualitative  in  nature,  and  even  though  the  T&E  Community  readily 
understands  that  many  system  attributes  are  qualitatively  evaluated,  the  study  team  determined 
that  quantifiable  examples  were  preferred  for  the  audience  that  the  study  team  intended  to 
reach.  This  was  the  audience  of  stakeholders  which  was  wont  to  assert  that  HSI-related 
requirements  were  not  testable. 

The  CTPs  selected  were,  for  the  most  part,  those  that  resulted  in  hard  numbers  being 
generated  from  accepted  measurement  methods  and  testing  devices.  Usability  CTPs  were 
included  in  multiple  examples  in  order  to  address  top  issue  number  five  from  the  workshop, 
“improve  usability  testing.”  Included  in  this  group  was  an  intuitive  user  interface  parameter 
which  required  comparison  of  human  performance  between  trained  and  untrained  users;  the 
interface  was  judged  intuitive  if  it  could  be  successfully  used  to  satisfy  mission  requirements  in 
the  absence  of  system-specific  training. 

Presentation  of  the  Examples 

Initial  versions  of  the  examples  were  indexed  by  HSI  domain  and  KPP.  These  appeared  in  two 
additional  left-hand  columns  not  found  in  Figure  2  (adding  columns  was  strictly  for  the  benefit  of 
comprehensibility  of  the  examples  -  there  is  no  recommendation  to  formally  change  the 
Framework  structure).  The  capability  (or  system)  type  was  not  included  in  the  presentation. 

Reviewers  found  it  difficult  to  make  sense  of  the  examples  in  the  absence  of  a  system  context. 
The  Framework  was  subsequently  modified  (again  for  the  purposes  of  these  examples  only)  to 
add  a  third  column  on  the  left  which  gave  the  system  type.  In  the  opinion  of  reviewers,  this 
increased  the  comprehensibility  of  the  examples. 

When  the  examples  were  shown  to  HSI  representatives,  there  was  disagreement  on  the  HSI 
domains  that  were  used  to  index  specific  examples.  The  study  team  believes  this  illustrates  the 
synergistic  quality  of  the  domains  which  are  relevant  to  definition,  but  perhaps  less  so  to 
practice.  There  is  a  good  deal  of  overlap  between  occupational  health  risk  and  human  factors 
designs,  for  example.  The  disagreements  on  attributed  domains  inhibited  review  of  the  rest  of 
the  table.  It  was  subsequently  postulated  that  removing  the  domains  and  indexing  the 
examples  by  system  type  would  facilitate  understanding  and  streamline  review.  This  inference 
turned  out  to  be  true. 
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The  set  of  examples  sent  out  to  the  communities  including  a  column  for  KPP/KSA  and  a  second 
column  for  system  type.  The  remainder  of  the  Framework  was  as  it  is  shown  in  the  notional 
example  in  the  figure  above. 

Feedback  on  the  Examples 

In  the  table  above,  HSI  relevance  to  JCIDS  KPPs  was  shown.  While  these  relationships  are 
demonstrable  from  the  definitions  of  the  KPPs  and  the  HSI  domains,  the  traditional 
interpretations  of  the  Survivability  and  Availability  KPPs,  as  examples,  do  not  include  HSI  as  the 
study  team  related  it.  Discussions  with  ASC/AQT  representatives  acknowledged  the 
relationship,  but  suggested  that  an  explanation  of  the  relevance  of  the  examples  provided  to  the 
KPPs  in  question  would  serve  to  make  those  examples  more  useful. 

It  should  be  emphasized  that  the  importance  of  the  measures  in  the  Survivability  and  Availability 
examples  was  not  questioned,  merely  the  names  of  the  KPPs  that  were  used  to  create  those 
examples.  These  examples  were  modified  to  include  footnotes  explaining  the  use  of  the  terms 
“Survivability”  and  “Availability”  in  this  context.  Additionally,  some  terms  used  as  KPPs  were 
thought  instead  to  be  KSAs  by  reviewers,  and  the  supported  decisions  in  the  final  column  were 
corrected  by  reviewers  in  some  cases. 

After  the  review  by  ASC/AQT,  all  of  the  examples  were  reordered  to  de-emphasize  the 
Survivability  and  Availability  examples  prior  to  sending  them  out  to  a  broader  T&E  audience. 

This  was  done  so  that  traditional  understanding  of  these  KPPs  would  not  cause  reviewers  to 
reject  them  out  of  hand.  KPPs  that  were  not  defined  as  “mandatory”  were  re-labeled  as  KSAs. 
Decisions  supported  were  reviewed  and  changed  as  recommended. 

Feedback  was  also  received  on  the  CTPs  and  test  methods  from  the  HSI  community. 
Representatives  of  the  AFFTC  rigorously  scrutinized  the  examples,  corrected  several  of  the 
CTPs  and  added  both  qualitative  and  quantitative  test  methods.  These  recommendations  were 
also  incorporated.  This  feedback  increased  the  study  team’s  confidence  in  the  examples. 

Value  of  the  Examples 

The  study  team  was  unable  to  discover  previously  completed  Framework  examples.  The  HSI 
Framework  examples  developed  by  the  study  team  may  be  the  first  to  exist  anywhere.  This 
situation  provides  an  opportunity  for  HSI  to  deeply  penetrate  the  T&E  Community.  This  was 
illustrated  by  the  ASC/AQT  representatives  who  were  encouraging  the  study  team  to  work  with 
training  and  community  of  practice  representatives  to  embed  the  examples  in  the  material  of 
those  organizations. 

Recommendations  and  Conclusions 

The  study  team  collected  numerous  and  diverse  findings  throughout  the  course  of  this  study. 
Many  of  the  findings  have  been  already  presented  throughout  this  report. 

•  The  study  team  took  the  findings  of  the  January  Workshop  and,  with  the  help  of  the 
Workshop  attendees,  was  able  to  develop  them  into  five  major  issues  (testable 
requirements;  guidance  documents;  linkages  with  existing  tests;  improved  measurement 
methods;  and  usability  shortfalls)  and  three  focus  areas. 

o  Interviews  with  T&E  and  HSI  SMEs  resulted  in  numerous  issues  that  continue  to 
need  attention  by  the  T&E  Community. 
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o  Review  of  primary  T&E  policy  and  guidance  resulted  in  nearly  200 

recommendations  for  specific  HSI  guidance  in  T&E  policy  as  well  as  “touch 
points”  for  HSI  awareness  while  executing  T&E  policy  (see  Appendices  D,  E,  and 
F). 

■  Each  T&E  policy  document  follows  a  rigorous  change  process.  SURVIAC 
recommends  continual  client  involvement  in  this  change  process  to 

ensure  the  recommendations  of  this  study  provide  the  optimal  HSI 

guidance  within  T&E  policy.  Barring  full  incorporation  of  the 
recommended  policy  changes.  SURVIAC  recommends  publishing  these 

recommended  changes  in  a  Guidebook.  In  addition.  SURVIAC  also 

recommends  the  development  of  a  suggested  HSI  template  for  AFPAM 

63-119  to  be  presented  to  AF/TEP. 

■  As  this  study  effort  was  unable  to  sufficiently  address  many  topic  areas 
sharing  common  ground  with  the  T&E  Community.  Such  topic  areas 
where  overlap  occurs  are:  foreign  materiel  testing,  modeling  &  simulation, 
stores  carriage  certification,  deficiency  reporting,  transition  to  production, 
OSS&E,  rapid  acguisition,  and  nuclear  weapons  life  cycle  management. 
SURVIAC  recommends  further  HSI  focus  on  these  topic  areas  to  ensure 

complete  coverage  of  HSI  in  these  areas  related  to  T&E. 
o  Development  of  HSI-specific  examples  for  the  new  T&E  Framework  was  able  to 
address  all  five  major  issues  identified  at  the  January  Workshop. 

The  study  team  also  concluded  there  were  three  themes  that  were  broader  than  the  initial  areas 
of  work,  January  Workshop  findings,  or  findings  from  the  three  focus  areas. 

•  HSI  Integration  with  T&E  Process:  The  Air  Force  T&E  process  is  distributed  over  a 
number  of  organizations.  While  there  is  a  centralized  policy  office,  application  of  that 
policy  is  decentralized.  Furthermore,  missions  vary  from  organization  to  organization, 
which  necessitate  differences  based  on  test  phase  (DT,  OT,  sustainment),  acguisition 
phase,  product  domain  (aircraft,  armament,  Command,  Control,  Communications, 
Computers,  Intelligence,  Surveillance,  Reconnaissance,  space),  and  other 
categorizations.  This  has  made  it  more  challenging  to  develop  a  standardized  approach 
to  integrating  HSI  in  T&E. 

o  SURVIAC  recommends  that  the  HSI  community  increase  its  network  of  contacts 

within  the  T&E  Community  at  every  opportunity.  This  will  include  not  only  links 
between  the  71 1th  HPW  and  test  organizations,  but  will  also  include  creating 
connections  between  HSI  domain  practitioners  and  the  T&E  Community. 

•  HSI  Awareness:  Awareness  of  HSI  within  the  T&E  Community  is  weak.  Most  personnel 
with  whom  the  study  team  interacted  had  heard  the  term  HSI,  but  eguating  HSI  with 
Human  Factors  Engineering  remains  prevalent.  The  T&E  Community,  both  AFMC  and 
SMC,  are  working  to  rebuild  their  workforce.  The  human  factors  presence,  once  a 
strong  link  to  HSI,  has  become  depleted  as  well.  Training  courses  are  being  developed 
and  modified  for  test  managers.  There  is  an  opportunity  to  incorporate  HSI  into  these 
courses. 

o  SURVIAC  recommends  supporting  AFHSIO  education  and  training  initiatives, 

some  which  are  already  ongoing,  to  include  more  HSI  in  T&E  training,  and  more 

T&E  in  HSI  training,  and  to  support  that  initiative  with  materials  and  findings  from 

this  study  as  reguired. 
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•  T&E  Framework:  The  recent  introduction  of  the  T&E  Framework  has  provided  HSI  with 
a  key  opportunity  to  address  many  of  the  issues  raised  at  the  January  Workshop 
(including  each  issue  on  the  Workshop  “Top  Five  List”).  It  is  necessary  to  remember 
that  while  these  T&E  Framework  examples  are  realistic  and  representative,  they  are  still 
contrived  and  cannot  be  over-generalized;  they  must  not  be  lifted  directly  into  a  TEMP 
without  significant  amounts  of  situational-specific  analysis. 

o  SURVIAC  shares  the  recommendation  of  this  project’s  government  sponsor  to 

create  a  library  of  Framework  examples  for  HSI  SMEs  to  use  as  guidance  while 

working  with  the  T&E  Community. 
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Appendix  B:  Glossary  of  Abbreviations  and  Acronyms 


71 1  HPW/HPOAir  Force  711th  Human  Performance  Wing/Human  Performance  Optimization  Division 
AF  Air  Force 

AF/TEP  Air  Force  Test  and  Evaluation  Directorate,  Policy  and  Programs  Division 

AFHSIO  Air  Force  Human  System  Integration  Office 

AFI  Air  Force  Instruction 

AFMAN  Air  Force  Manual 

AFMC  Air  Force  Materiel  Command 

AFPAM  Air  Force  Pamphlet 

AIAA  American  Institute  of  Aeronautics  and  Astronautics 

ANSI  American  National  Standards  Institute 

ASC  Aeronautical  Systems  Center 

ASC/AQT  Aeronautical  Systems  Center/Center  Test  Authority 

ODD  Capabilities  Development  Document 

CDRL  Contract  Data  Requirements  List 

CLIN  Contract  Line  Item  Number 

COI  Critical  Operational  Issue 

CPD  Capabilities  Production  Document 

CT  A  Center  T  est  Authority 

CTP  Critical  Test  Parameter 

DAG  Defense  Acquisition  Guidebook 

DoD  Department  of  Defense 


DoDI 

DR 

DRB 

ESOH 

HFE 

HPW 

HSI 

ICD 

IOT&E 

ITT 

JCIDS 

KSA 

KPP 

MOE 

MOS 

OTA 

OV 

PEO 

SME 


Department  of  Defense  Instruction 

Deficiency  Report  /  Deficiency  Reporting 

Deficiency  Review  Board 

Environment,  Safety,  and  Occupational  Health 

Human  Factors  Engineering 

Human  Performance  Wing 

Human  Systems  Integration 

Initial  Capabilities  Document 

Initial  Operational  Test  and  Evaluation 

Integrated  Test  Team 

Joint  Capabilities  Integration  and  Development  System 

Key  System  Attribute 

Key  Performance  Parameter 

Measure  of  Effectiveness 

Measure  of  Suitability 

Operational  Test  Agency 

Operational  View 

Program  Executive  Officer 

Subject  Matter  Expert 
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SURVIAC 

TAG 

T&E 

TEMP 

TES 

T.O. 


Survivability  and  Vulnerability  Information  Analysis  Center 

Technical  Advisory  Group 

Test  and  Evaluation 

Test  and  Evaluation  Master  Plan 

Test  and  Evaluation  Strategy 

Technical  Order 
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Appendix  C:  Interview  Discussion  Guide 

The  Air  Force  Human  Systems  Integration  Office  and  the  711th  Human  Performance  Wing  are  interested 
in  the  amount  of  interaction  of  human  systems  integration  (HSI)  in  the  DoD  acquisition  cycle.  The 
following  questions  specifically  deal  with  the  interaction  of  HSI  with  test  &  evaluation  (T&E).  Unless 
otherwise  stated,  please  relate  these  questions  to  your  current  work  assignment. 

YOUR  HSI-in-T&E  BACKGROUND 

What  is  your  primary  role  in  the  T&E  process? 

a.  Capability  Requirements  Definition 

b.  T&E  Policy 

c.  Verification  Planning  -  during  requirements  development 

d.  T&E  Planning 

e.  T&E  Execution 

f.  T&E  Verification  and  Validation 

g.  Other: _ 


Please  briefly  describe  your  primary  T&E  responsibilities: 


In  which  weapon  systems  domain  do  you  primarily  work? 


a. 

Aeronautics 

Technology  Dev. 

□ 

Product 

□ 

Logistics 

□ 

b. 

Armament 

Technology  Dev. 

□ 

Product 

□ 

Logistics 

□ 

c. 

C2ISR 

Technology  Dev. 

□ 

Product 

□ 

Logistics 

□ 

d. 

Space 

Technology  Dev. 

□ 

Product 

□ 

Logistics 

□ 

e. 

Nuclear  Weapons 

□ 

Product 

□ 

Logistics 

□ 

Technology  Dev. 

f.  Personnel  Equipment*  □  Product  □ 

Logistics  □  Technology  Dev. 

g.  Other: _ 

*e.g.,  night  vision  goggles,  etc. 


If  you  are  an  HSI  professional,  in  which  domains  do  you  routinely  work?  (check  all  that  apply) 

□  Manpower 

□  Personnel 

□  Training 

□  Environment 
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□  Safety 

□  Occupational  Health 

□  Human  Factors  Engineering 

□  Habitability 

□  Survivability 

□  Others? 


How  do  you  reach  out  to  the  HSI  domains  in  which  you  have  less  expertise? 


In  your  experience,  who  brings  the  considerations  of  the  domains  together? 


•  What  is  the  process  for  making  this  happen? 


If  you  are  not  an  HSI  professional,  with  which  HSI  domains  do  you  routinely  work?  (check  all  that  apply) 

□  Manpower 

□  Personnel 

□  Training 

□  Environment 

□  Safety 

□  Occupational  Health 

□  Human  Factors  Engineering 

□  Habitability 

□  Survivability 

□  Others? _ 


•  How  do  you  reach  out  to  the  other  HSI  domains? 

In  your  experience,  who  brings  the  considerations  of  the  domains  brought  together? 

•  What  is  the  process  for  making  this  happen? 

How  does  including  HSI  in  test  contribute  to  confidence  in  a  system's  mission  effectiveness? 


YOUR  ROLE  IN  TESTABLE  REQUIREMENTS  DEVELOPMENT 

Are  you  currently,  or  have  you  in  the  past  been,  involved  with  requirements  development  for 

□  Initial  capability  documents  (ICDs)? 
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□  Capability  development  documents  (CDDs)? 

□  Capability  production  documents  (CPDs)? 

□  System  Requirements  Documents  (SRDs)? 

□  Requirements  Development? 

□  Documenting  Verification  and  Validation  Approaches? 

□  Source  Selection  Reviews? 

□  Review  of  Contractor-Developed  Specs? 

□  Participation  in  Design  Reviews? 

□  Others? _ 


What  resources  do  you  routinely  use  when  developing  requirements? 

•  Specific  Documents? 

•  Specific  Web  Sites? 

•  Specific  Organizations? 

•  Specific  Personnel? 

•  Other? 


Have  you  ever  served  on  a  High 

Performance  Team  (HPT)?  □  Yes  □  No  □  Don't  know 

what  an  HPT  is. 

o  If  yes,  what  mechanisms  did  you  use  to  "reach  back"  to  others  in  your  profession? 


•  Are  you  a  certified 

requirements  manager?  □  Yes  □  No 

i.e.,  Passed  DAU  CLM  041  and  RQM  101? 

•  Have  you  ever  known 

of  a  T&E  professional  who  has  supported  an  HPT?  □  Yes  □  No 

o  If  yes,  what  mechanisms  did  they  use  to  "reach  back"  to  other  T&E  professionals? 
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•  Have  you  ever  known 

of  a  HSI  professional  who  has  supported  an  HPT?  □  Yes  □  No 

o  If  yes,  what  mechanisms  did  they  use  to  "reach  back"  to  other  HSI  professionals? 


What  do  you  believe  is  the  most  appropriate  role  for  an  HSI  professional  on  an  HPT? 


How  do  you  believe  the  T&E  professional  on  an  HPT  should  reach  out  to  HSI  domain  experts? 


How  do  you  believe  the  HSI  professional  on  an  HPT  should  reach  out  to  other  HSI  domain  experts? 


Example  for  Discussion  Leaders: 

o  Well-defined  requirements  are: 

■  Specific 

■  Measurable 

■  Achievable 

■  Traceable  to  user  need 

■  Testable 


•  How  can  well  defined  and  testable  HSI  requirements  be  included  in  ICDs,  CDDs,  or  CPDs? 


•  Mandatory  KPPs,  as  defined  in  the  July  2009  MANUAL  FOR  THE  OPERATION  OF  THE  JOINT 
CAPABILITIES  INTEGRATION  AND  DEVELOPMENT  SYSTEM  (JCIDS),  are  listed  below, 
o  Survivability 
o  Force  Protection 
o  Sustainment 

■  Availability 

■  Reliability 

■  Ownership  Cost 
o  Net  Ready 

o  System  Training 
o  Energy  Efficiency 


•  Have  you  seen  mandatory  KPPs  used  to  motivate  HSI  in  requirements  documents? 
Mandatory  KPP 

Survivability?  □  Yes  □  No  How?  _ 
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Force  Protection?  □ 

Sustainment?  □ 
Availability?  □ 
Reliability?  □ 
Ownership  Cost?  □ 

Net  Ready?  □ 

System  Training?  □ 
Energy  Efficiency?  □ 
Others?  _ 


Yes 

□ 

No 

How? 

Yes 

□ 

No 

How? 

Yes 

□ 

No 

How? 

Yes 

□ 

No 

How? 

Yes 

□ 

No 

How? 

Yes 

□ 

No 

How? 

Yes 

□ 

No 

How? 

Yes 

□ 

No 

How? 

•  Have  you  seen  cases  when  HSI  requirements  were  motivated  by  -  KPPs  that  are  not  on  the 
mandatory  list?  □  Yes  □  No 

How? _ 


•  What  is  the  relationship  between  KPPs,  KSAs,  APAs  and  critical  operational  issues  (COIs)?  For 
example,  are  KPPs  used  directly  to  develop  COIs? 


•  How  can  HSI  be  included  in  critical  operational  issues,  measures  of  effectiveness,  measures  of 
suitability,  and  measures  of  performance? 


•  What  is  the  relationship  between  measures  of  effectiveness,  measures  of  suitability,  and  measures 
of  performance  and  critical  test  parameters?  For  example,  does  each  measure  of  effectiveness  map 
directly  to  a  critical  test  parameters?  multiple  critical  test  parameters? 


•  How  do  you  personally  review  candidate  requirements  to  assure  they  are  testable?  What  do  you 
look  for?  Think  about? 


Many  HSI  requirements  are  derived  from  "operating  concepts".  What  is  your  understanding  of  what  an 
"operating  concept"  is? 


How  do  you  obtain  "operating  concepts"?  Who  creates  them? 
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Are  DoDAF  architecture  products,  such  as  the  High-Level  Operational  Concept  Graphic  (OV-1)  or  the 
Operational  Activity  Model  (OV-5b)  used  to  support  test  requirements  development? 


Illustrative  example  for  discussion  leaders  to  use  if  needed. 


Some  HSI  requirements  are  dismissed  by  HPT  members  because  they're  not  considered  testable.  HPT 
members  seem  to  be  unfamiliar  with  human  systems  test  and  measurement  techniques.  Can  you  think 
of  a  way  this  could  be  addressed? 


o  What  test  measures  would  you  recommend  to  satisfy  these  concerns? 


What  specific  programs  (name,  series)  with  which  you're  familiar  did  a  good  job  of  crafting  testable  HSI 
requirements? 


ALIGNING  HSI  WITH  T&E 

How  are  HSI  people  contacted  for  support  to  T&E  activities? 


•  Given  the  breadth  of  the  nine  HSI  domains,  how  are  you  assured  you  will  have  the  "right"  HSI  person 
for  each  task? 


What  qualifications  should  an  HSI  professional  possess  in  order  to  successfully  participate  in  the  T&E 
process? 


What  techniques  should  be  used  to  maintain  the  "visibility"  of  HSI  throughout  the  life  cycle? 


TEST  PLAN  DOCUMENTATION  -  STRATEGIES  AND  MASTER  PLANS 

Are  you  currently  or  have  you  in  the  past  been  involved  with  developing  □  Yes  □  No 
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test  strategy  and  test  master  planning? 

What  resources  do  you  routinely  use  when  developing  strategies  and  master  plans? 

•  Specific  Documents? 

•  Specific  Web  Sites? 

•  Specific  Organizations? 

•  Specific  Personnel? 

•  Other? 

Have  you  ever  served  on  an  Integrated  Test  Team  (ITT)?  □  Yes  □  No 

If  yes,  what  mechanisms  did  you  use  to  "reach  back"  to  others  in  your  profession? 

Have  you  ever  known  of  a  T&E  professional  who  has  served  on  an  ITT?  □  Yes  □  No 

If  yes,  what  mechanisms  did  they  use  to  "reach  back"  to  other  T&E  professionals? 

Have  you  ever  known  of  an  HSI  professional  who  has  served  on  an  ITT?  □  Yes  □  No 

If  yes,  what  mechanisms  did  they  use  to  "reach  back"  to  other  HSI  professionals? 

What  do  you  believe  is  the  most  appropriate  role  for  an  HSI  professional  on  an  ITT? 

How  do  you  believe  the  HSI  professional  on  an  ITT  should  reach  out  to  other  HSI  domain  experts? 

Which  HSI  tests  should  be  performed  in  Developmental  Test  &  Evaluation  (DT&E)? 

What  HSI  tests  can  be  performed  in  Operational  Test  &  Evaluation  (OT&E)? 

What  are  the  constraints? 
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How  do  HSI  tests  break  out  between  tests  of  mission  effectiveness  and  logistics  tests? 

In  which  tests  are  HSI  attributes  customarily  included? 

Which  HSI  attributes  are  generally  tested  in  performance  tests? 

Which  HSI  attributes  are  generally  tested  in  logistics  tests? 

What  programs  with  which  you're  familiar  did  a  good  job  of  crafting  strategies  and  master  plans  that 
effectively  incorporated  HSI  requirements  in  big-picture  testing  plans? 

TEST  PLAN  DOCUMENTATION  -  TEST  EXECUTION  PLANS  AND  PROCEDURES 

Are  you  currently  or  have  you  in  the  past  been  involved  with  developing  □  Yes  □  No 

test  plans  and  procedures? 

What  resources  do  you  routinely  use  when  developing  test  plans  and  procedures? 

•  Specific  Documents? 

•  Specific  Web  Sites? 

•  Specific  Organizations? 

•  Specific  Personnel? 

•  Other? 

Have  you  successfully  included  HSI  requirements  into  test  plans  and  procedures?  □  Yes  □  No 

Have  you  successfully  included  HSI  metrics  into  test  plans  and  procedures?  □  Yes  □  No 

Have  you  interacted  with  human  factors  engineers  in  developing  test  plans  □  Yes  □  No 

and  procedures? 

What  do  you  believe  is  the  most  appropriate  role  for  an  HSI  professional  in  developing  test  plans  and 
procedures? 
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How  do  you  believe  the  T&E  professional  involved  in  developing  test  plans  and  procedures  should  reach 
out  to  HSI  domain  experts? 


How  do  you  believe  the  HSI  professional  who  is  involved  in  developing  test  plans  and  procedures  should 
reach  out  to  other  HSI  domain  experts? 


Does  HSI  T&E  ever  occur  before  milestone  B? 


What  language  have  you  used  to  specify  test  criteria  so  the  human  is  included  (human-in-the-test-loop) 
in  the  performance  requirement  being  tested  (e.g.,  timing  a  test  to  include  user  understanding  of 
situation  and  subsequent  execution  of  keystroke)? 


Do  you  use  or  know  of  a  resource  or  repository  that  lists  and  describes  tests  □  Yes  □  No 
related  to  human  performance  and  suitability? 


What  resources  exist? 


What  objective  HSI  measures  can  be  collected  during  flight  test  (DT&E  or  OT&E)? 


Have  you  ever  used  the  SWOT  method  (Strengths,  Weaknesses,  □  Yes  □  No  □  Never 
Opportunities,  and  Threats)  for  workload  measures?  heard  of 


Have  you  applied  SWOT  results  to  test?  □  Yes  □  No 

•  DT&E?  □  Yes  □  No 

•  OT&E?  □  Yes  □  No 

Are  you  familiar  with  the  Design  of  Experiments  (DOE)  process?  □  Yes  □  No  □  Never 

heard  of 


Have  you  ever  participated  in  a  DOE  analysis?  □  Yes  □  No 

Was  the  DOE  related  to  test  design?  □  Yes  □  No 


How  is  logistics  suitability  evaluated? 
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How  can  HSI  attributes  of  cross-platform  interoperability  (e.g.,  usability)  be  measured? 


What  programs  with  which  you're  familiar  did  a  good  job  of  crafting  execution  test  plans  and 
procedures  that  effectively  incorporated  HSI  requirements  in  big-picture  testing  plans? 


TEST  EXECUTION 

Are  you  currently  or  have  you  in  the  past  been  involved  in  conducting  tests  □  Yes  □  No 

and/or  analysis  of  test  data? 

How  would  you  generally  characterize  the  types  of  T&E  you  perform? 

Have  you  interacted  with  human  factors  engineers  in  conducting  tests  and/or  □  Yes  □  No 

analysis  of  test  data? 


What  do  you  believe  is  the  most  appropriate  role  for  an  HSI  professional  involved  in  test  execution 
and/or  analysis  of  test  data? 


How  do  you  believe  the  T&E  professional  involved  in  test  execution  and/or  analysis  of  test  data  should 
reach  out  to  HSI  domain  experts? 

How  do  you  believe  the  HSI  professional  involved  in  test  execution  and/or  analysis  of  test  data  should 
reach  out  to  other  HSI  domain  experts? 


What  qualifications  should  testers  have  in  order  to  successfully  conduct  tests  with  HSI  implications? 


If  users  are  used  to  evaluate  HSI  implementations  (e.g.,  human-system  interfaces),  how  have  you 
avoided  having  the  design  being  skewed  by  single-user  preferences? 


How  are  human  test  participants  selected? 


What  processes  are  used  to  qualify  system  operators  as  test  subjects? 


What  processes  are  used  to  qualify  system  maintainers  as  test  subjects? 


What  infrastructure  tools  are  missing  for  human  performance  measurement? 
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•  In  DT&E? 


•  In  OT&E? 


Have  you  ever  used  small  end  statistical  methods?  □  Yes  □  No  □  Never 

heard  of 


When? 


How  do  you  factor  system  safety  (what  has  been  designed  into  the  system)  into  test  planning  and 
execution? 


Do  HSI  concerns  factor  into  the  OT&E  readiness  certification? 

□ 

Yes 

□ 

No 

Are  HSI  reps  ever  involved? 

□ 

Yes 

□ 

No 

TEST  REPORTS  AND  DEFICIENCY  REPORTING 

Where  are  HSI-related  test  results  documented? 


Are  you  currently  or  have  you  in  the  past  been  involved  with  □  Yes  □  No 

deficiency  reporting? 


How  can  users  be  included  in  acceptance  review  of  watch  list  items  prior  to  conversion  to  deficiency 
reports? 

How  can  HSI  be  considered  during  Deficiency  Review  Board  (DRB)  deliberations? 

What  do  you  believe  is  the  most  appropriate  role  for  an  HSI  professional  on  a  DRB? 

How  do  you  believe  the  T&E  professional  on  a  DRB  should  reach  out  to  HSI  domain  experts? 


o  How  do  you  believe  the  HSI  professional  on  a  DRB  should  reach  out  to  other  HSI  domain 
experts? 
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CONTRACTOR  DOCUMENTATION 


Have  you  ever  supported  definition  of  contractor  documentation? 

□ 

Yes 

□ 

No 

Request  for  Proposal  (RFP)? 

□ 

Yes 

□ 

No 

Statement  of  Work  (SOW)? 

□ 

Yes 

□ 

No 

Data  Information  Documents  (DIDs)? 

□ 

Yes 

□ 

No 

Contract  Data  Requirements  Lists  (CDRLS)? 

□ 

Yes 

□ 

No 

Contract  Line  Item  Numbers  (CLINS)? 

□ 

Yes 

□ 

No 

System  Requirements  Document  (SRD)? 

□ 

Yes 

□ 

No 

Have  you  been  successful  in  inserting  HSI  considerations  into  these  documents 

□ 

Yes 

□ 

No 

Why? 

Can  you  think  of  programs  which  did  a  good  job  of  incorporating  HSI  into 

□ 

Yes 

□ 

No 

contractor  documents? 

Please  provide  program  names  if  possible. 

Name  some  ways  in  which  you  think  HSI  incorporation  in  contractor  test  could  be  improved. 

PROCESS  IMPROVEMENT 

If  you  can,  name  a  mechanism,  repository  or  tool  that  would  provide  the  HSI  community  with  insight 
into  human  interface  issues  that  are  being  found  on  systems  during  development  or  operations? 
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Appendix  D:  Recommendations  to  AFI  99-103,  Capabilities-Based  Test  and 
Evaluation 

This  appendix  contains  recommendations  for  modifications  to  AFI  99-103.  Recommended  additions  or  modifications  are  in  red. 
Because  the  study  sought  interaction  points  between  the  HSI  and  T&E  processes,  the  “HSI  Touch  Point”  column  is  included  to 
indicate  HSI  activities  associated  with  those  described  in  the  AFI  that  would  improve  HSI’s  treatment  in  T&E  or,  more  generally,  in 
acquisition.  Red  indicates  additions  to  existing  text. 


Section 

Section  Title 

As  Is 

Recommended 

HSI  Touch  Point 

1.1 

Purpose  of 

Test  and 
Evaluation 

HSI  is  strongly  related  to  OSS&E 
(mission  capability).  Incorporate 
HSI  into  T&E  to  identify  HSI- 
related  risks. 

1.1.1 

- 

- 

- 

Collaborate  with  T&E  personnel. 

1.1.2 

" 

“ 

“ 

Provide  test  results  to  HSI  Status 
Board. 

1.1.3 

Help  manage  risks  during 
engineering,  acquisition,  fielding, 
and  sustainment  by  accurately 
characterizing  system  technical 
and  operational  performance 
throughout  the  system  life  cycle. 

Help  manage  risks  during 
engineering,  acquisition,  fielding, 
and  sustainment  by  accurately 
characterizing  system  technical, 
human  and  operational 
performance  throughout  the 
system  life  cycle. 

1.1.4 

Help  the  acquisition  and 
sustainment  communities 
acquire  and  maintain 
operationally  effective,  suitable, 
survivable,  and  secure  systems 
for  Air  Force  operators. 

Help  the  acquisition  and 
sustainment  communities 
acquire  and  maintain 
operationally  effective,  suitable, 
safe,  survivable,  and  secure 
systems  for  Air  Force  operators. 

1.1.5 

- 

- 

- 

Support  refinement  of  Tactics, 
Techniques  and  Procedures. 
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Section 

Section  Title 

As  Is 

Recommended 

HSI  Touch  Point 

1.2.2 

Collaborative 
Concepts  and 
Processes 

In  order  to  be  able  to  impact 
“Left-of-A”,  AFI  10-601  and  the 
JCIDS  Process  must  be 
impacted. 

1.3 

Seamless 

Verification 

Integrate  HSI  tests  with  other 

T&E  efforts  via  the  TES,  TEMP 
and  test  plans.  Ensure  T&E 
databases  are  configured  to 
include  HSI-related  parameters. 

1.3.2 

Refocuses  T&E  of  materiel 
solutions  on  capabilities-based 
requirements  and  operational 
mission  needs  instead  of 
traditional  pass-fail 
measurements  of  specification¬ 
like  requirements. 

Refocuses  T&E  of  materiel 
solutions,  including  the  human 
elements,  on  capabilities-based 
requirements  and  operational 
mission  needs  instead  of 
traditional  pass-fail 
measurements  of  specification¬ 
like  requirements. 

Help  to  define  operating 
concepts  for  systems. 

1.4 

Integrated 

Test  Team 

ITT  membership  will  include 
representatives  from  the 
responsible  test  organization 
(RTO),  operational  test 
organizations,  participating  test 
organizations  (PTO),  system 
contractors,  and  the  acquisition, 
requirements,  intelligence, 
operations,  and  support 
communities. 

ITT  membership  will  include 
representatives  from  the 
responsible  test  organization 
(RTO),  operational  test 
organizations,  participating  test 
organizations  (PTO),  system 
contractors,  and  the  acquisition, 
requirements,  Human  Systems 
Integration  (HSI),  intelligence, 
operations,  and  support 
communities. 

Have  HSI  representation  on  ITT. 
Set  up  a  reachback  network  for 

ITT  HSI  rep  to  enable  lifeline 
contact  with  domain  specialists 
for  detail  questions. 

1.5.1 

Tailoring 

Contribute  to  identify  operators’ 
requirements  prior  to  the  Material 
Development  Decision. 

1.5.3 

Early 

Deficiency 

Resolution 

Contribute  to  identification  of 
human-related  deficiencies. 
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1.5.5 

Inclusion  of 

HSI  in 
Operational 
Suitability 
Evaluations 

Operational  Suitability 
evaluations  should  include  HSI 
attributes  (environment,  safety, 
occupational  health,  human 
factors,  usability,  habitability, 
manpower,  personnel,  natural 
environment  effects  and 
impacts,  and  training)  as 
appropriate,  in  accordance  with 
AFI  10-601,  when  determining 
the  degree  to  which  a  system 
can  be  placed  and  sustained 
satisfactorily  in  field  use. 

2.2.1 

Assesses  the  technological 
capabilities  of  systems  or 
concepts  in  support  of 
requirements  activities  described 
in  AFI  10-601  ... 

Assesses  the  combined 
technological  and  human 
capabilities  of  systems  or 
concepts  in  support  of 
requirements  activities  described 
in  AFI  10-601  ... 

The  usefulness, 
understandability,  usability, 
safety  and  suitability  of  the 
capabilities  must  be  assessed. 

2.2.7 

Characterizes  system 
performance,  military  utility,  and 
determines  system  safety. 

Characterizes  system 
performance,  military  utility,  and 
determines  system  safety  and 
usability. 

Usability  is  a  key  element  of 
operational  suitability.  It  supports 
capability  availability. 

2.2.10 

OSS&E  has  a  strong  relation  to 
HSI.  Systems  that  are  suitable 
conform  to  user  needs.  Safety  is 
an  HSI  key  interest.  Effective 
systems  are  ones  that  enable 
forces  and  warfighters  to  meet 
operational  goals. 
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2.2.11 

As  with  OSS&E,  Reliability, 
Availability  and  Maintainability 
(RAM)  has  a  strong  relation  to 

HSI.  Reliable  systems  are  those 
that  do  not  induce  human  errors. 
System  availability  is  improved 
when  human  safety,  occupational 
health,  protection  and  survival 
are  assured.  Maintainability  is 
another  facet  of  usability  and 
understandability. 

2.4.1 

Provide  information  to  decision 
makers  on  potential  operator 
casualties,  system 
vulnerabilities,  lethality,  and 
system  recoverability  while 
taking  into  equal  consideration 
the  susceptibility  to  attack  and 
combat  performance  of  the 
system. 

Provide  information  to  decision 
makers  on  potential  operator 
casualties,  operator  survivability, 
force  protection,  system 
vulnerabilities,  lethality,  and 
system  recoverability  while 
taking  into  equal  consideration 
the  susceptibility  to  attack  and 
combat  performance  of  the 
system. 

Addresses  two  KPPs  that  relate 
to  HSI. 

2.4.2 

HSI  analysts  help  to  define 
realistic  conditions  from  the 
human  standpoint. 

2.4.3 

“Repair  capabilities”  have 
usability  implications  just  as 
maintainability  does. 
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2.5 

Operational 

Testing 

Operational  testing  may  also 
look  at  doctrine,  operational 
concepts  (as  described  in  AFPD 
10-28,  Air  Force  Operational 
Concepts),  system  performance, 
TTPs,  logistics  support 
elements,  intelligence  support 
elements,  system  interoperability 
and  security,  materiel  issues, 
safety,  training,  organization, 
human  systems  integration 
(HSI),  and  personnel. 

Operational  testing  may  also 
look  at  doctrine,  operational 
concepts  (as  described  in  AFPD 
10-28,  Air  Force  Operational 
Concepts),  system  performance, 
TTPs,  logistics  support 
elements,  intelligence  support 
elements,  system  interoperability 
and  security,  materiel  issues, 
usability,  safety,  training, 
organization,  human  systems 
integration  (HSI),  and  personnel. 

Each  of  the  listed  items  has 
human  dimensions. 

2.6.1 

Initial 

Operational 
Test  and 
Evaluation 

AFOTEC  determines  the 
operational  effectiveness  and 
suitability  of  the  items  under 
test... 

AFOTEC  determines  the 
operational  effectiveness,  safety 
and  suitability  of  the  items  under 
test... 

HSI  analysts  help  to  characterize 
representative  personnel. 

2.6.9 

Operational 

Assessment 

They  provide  early  operational 
data  and  feedback  from  actual 
testing  to  developers,  operators, 
and  decision  makers. 

They  provide  early  operational 
data  and  feedback  from  actual 
testing  to  developers,  operators, 
sustainers,  and  decision  makers. 

2. 6. 9.3 

Identify  deficiencies  or  design 
problems  that  can  impact 
system  capability  to  meet 
operational  requirements,  the 
mission,  and/or  employment 
concepts. 

Identify  deficiencies  or  design 
problems  including  operator  and 
maintainer  workarounds  that  can 
impact  system  capability  to  meet 
operational  requirements,  the 
mission,  and/or  employment 
concepts. 

Workarounds  are  clear  sign  that 
the  user  interface  is  inadequate 
to  satisfy  requirements  and 
operational  needs.  Workarounds 
may  also  be  responses  to 
modified  operating  concepts  or 
new  threat  environments  that 
need  to  be  incorporated  into 
product  definitions  (see  2. 6. 9. 4). 

2. 6. 9.5 

Prototype  demonstrations 
provide  opportunities  to  improve 
user  interfaces. 
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2.6.10 

Early 

Operational 

Assessments 

EOAs  are  similar  to  OAs,  except 
they  are  performed  prior  to  MS  B 
to  provide  very  early 
assessments  of  system 
capabilities  and  programmatic 
risks. 

EOAs  are  similar  to  OAs,  except 
they  are  performed  prior  to  MS  B 
to  provide  very  early 
assessments  of  system 
capabilities,  aed-programmatic 
risks,  and  OSS&E. 

2.6.12 

Sufficiency  of 
Operational 
Test  Review 

...The  SOTR  may  not  be  used 
for  acquisition  milestone 
decisions  associated  with  OSD 
OT&E  Oversight  programs 
unless  approved  by  DOT&E. 

See  paragraph  7.5.5. 

...The  SOTR  may  not  be  used 
for  acquisition  milestone 
decisions  associated  with  OSD 
OT&E  Oversight  programs 
unless  approved  by  DOT&E. 
SOTRs  should  include  any 
human-related  findings  and 
conclusions  as  appropriate.  See 
paragraph  7.5.5. 

2.9 

Joint  Test  and 

The  JT&E  Program  focuses  on 

The  JT&E  Program  focuses  on 

Non-materiel  capabilities 

Evaluation 

evaluating  current  equipment, 
organizations,  threats,  doctrine, 
TTPs,  test  methodologies,  and 
system  interoperability  in 
realistic  environments. 

evaluating  current  equipment, 
organizations,  team  design, 
usability,  threats,  doctrine,  TTPs, 
test  methodologies,  and  system 
interoperability  in  realistic 
environments. 

nevertheless  have  implications  to 
workflow  and  the  usefulness  and 
usability  of  the  capability;  HSI 
contributes  to  a  complete 
evaluation. 

3.4.6 

" 

- 

Ensure  that  HSI  is  addressed  in 
test  planning. 

3.5.6 

- 

- 

Ensure  that  HSI  is  addressed  in 
test  planning. 

Section  parallels  3.4,  but  for 
space  systems. 

3.8.6 

“ 

“ 

“ 

Points  toAFI  16-1002  -  M&S 
must  consider  all  HSI  aspects. 

3.8.14 

Points  to  AFI  63-104  -  all 
aspects  of  HSI  should  be 
considered  within  the  SEEK 
EAGLE  program. 

3.8.16 

" 

- 

- 

Deficiency  reporting  process 
must  take  HSI  into  consideration. 
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4.2 

Early  Tester 
Involvement  in 
Requirements 
Development 

Testers  can  help  to  ensure  HSI- 
related  requirements  are  not 
thrown  out  by  the  HPT  for  being 
perceived  as  “untestable.” 

4.4.3 

ITT 

Membership 

The  ITT  should  include  expertise 
from  organizations  such  as  the 
SPO  (or  initial  SPO  cadre), 
AFOTEC,  and/or  MAJCOM 
operational  tester  (as 
appropriate),  SAF/AQ  or 

SAF/US,  HQ  USAF/TE,  HQ 
USAF/A3/5,  SAF/XC,  JITC, 

OSD,  ALCs,  product  centers, 
contractor,  developer,  science 
and  technology,  intelligence, 
developmental  testers, 
requirements  sponsors,  test 
facilities,  and  other  stakeholders 
as  needed  during  various  test 
program  phases. 

The  ITT  should  include  expertise 
from  organizations  such  as  the 
SPO  (or  initial  SPO  cadre), 
AFOTEC,  and/or  MAJCOM 
operational  tester  (as 
appropriate),  SAF/AQ  or 

SAF/US,  HQ  USAF/TE,  HQ 
USAF/A3/5,  SAF/XC,  JITC, 

OSD,  ALCs,  product  centers, 
contractor,  developer,  science 
and  technology,  intelligence, 
developmental  testers, 
requirements  sponsors,  test 
facilities,  HSI,  and  other 
stakeholders  as  needed  during 
various  test  program  phases. 

ITTs  have  been  identified  by  the 
T&E  community  a  high-payoff 
activities  for  incorporating  human 
considerations  into  system 
performance  testing. 

5.3 

Critical  Test 
Parameters 

CTPs  should  be  developed  to 
address  all  major  areas  of 
system  performance  and  should 
correlate  to  all  key  requirement 
areas,  to  include  all  KPPs. 

CTPs  should  be  developed  to 
address  all  major  areas  of 
system  performance,  including 
human,  and  should  correlate  to 
all  key  requirement  areas,  to 
include  all  KPPs. 

While  personnel  are  mentioned 
in  the  following  list,  explicit 
inclusion  of  humans  removes 
techno-centric  bias  that  may  exist 
in  planning,  and  supports  the 
total  system  approach  called  out 
in  5000  series  documents. 

5.5 

Common  T&E 
Data 

Management 

Operational  testers  must  allow 
open  data  sharing  and  non¬ 
interference  observation  by  all 
other  testers,  the  system 
developer,  contractor,  operators, 
DOT&E,  and  the  PM. 

Operational  testers  must  allow 
open  data  sharing  and  non¬ 
interference  observation  by  all 
other  testers,  the  system 
developer,  contractor,  operators, 
DOT&E,  HSI  personnel,  and  the 
PM. 

HSI-related  data  in  the  common 
database  should  be  severable  to 
support  evidence-based  practice. 
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5.10.8 

Warfighter 

Survivability 

HSI  engineers  shall  be  involved 
in  the  planning  for,  conduction  of 
and  evaluation  of  warfighter 
survivability  assessments. 

5.11.1 

Early 

Operational 

Assessments 

EOAs  are  opportunities  to  assess 
usability,  understandability  and 
usefulness. 

5.11.2 

Operational 

Assessments 

OAs  are  opportunities  to  assess 
the  human  considerations  of 
OSS&E. 

5.14.2.1 

SAF/AQ  AFHSIO  should  be  on 
the  TEMP  coordination  list. 

(Review  99-103,  Attachment  2, 
Information  [coordination] 
Requirements.) 

5.14.2.2 

- 

“ 

“ 

SAF/AQ  AFHSIO  should  be  on 
the  TEMP  coordination  list. 

5.14.2.3 

An  HSI  cell  representative  should 
support  the  Service  Acquisition 
Executive  or  designated 
representative  in  the  TEMP 
approval  process. 

5.14.3 

Multi-Service 

TEMPs 

The  lead  service  should  ensure 
that  its  HSI  office  has 
coordinated  on  the  TEMP. 

6.3.1 

HSI  engineers  must  make  sure 
COIs,  MOEs  and  MOSs  capture 
human  considerations  related  to 
operational  capabilities  and 
CONOPS. 

6.4 

Realistic 

Testing 

HSI  analysts  should  participate  in 
test  scenario  development,  so 
they  are  realistic  from  a  human 
perspective. 
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6.5 

Integrated 
Technical  and 
Safety 

Reviews 

Safety  integration  is  part  of  total 
life  cycle  systems  management 
per  DODD  5000.01  section 

El. 1.29. 

6.5.2 

Safety  Review 
Board 

Environmental  safety  and 
occupational  health  hazard 
mitigation  is  part  of  total  life  cycle 
systems  management  per  DODD 
5000.01  section  El.  1.29. 

6.7.1 

Operational 
Test  Concept 
Briefings 

SAF/AQ  HSIO  should  be  part  of 

Air  Force-level  reviews  and  multi¬ 
service  review  teams. 

6.7.2 

Operational 
Test  Plans 
and  Test  Plan 
Briefings 

SAF/AQ  HSIO  should  be  part  of 

Air  Force-level  reviews  and  multi¬ 
service  review  teams. 

6.9.1 

Joint 

Reliability  and 
Maintainability 
Evaluation 
Team 

HSI  needs  to  be  represented  on 
the  Joint  Reliability  and 
Maintainability  Evaluation  Team 
because  of  the  linkage  between 
RAM  and  human  safety  and 
performance. 

6.10 

Deficiency 

Reports 

HSI  needs  to  have  a  voice  in 
making  the  “nice  to  have” 
decision  as  the  designation  may 
have  manpower,  personnel  or 
training  cost  implications. 

6.10.4 

Tracking  and 
Closing  DRs 

" 

" 

HSI  should  participate  in  the 
decision  to  close  a  DR. 

6.13.1 

HSI  should  participate  in 
Capabilities  and  Limitations 
(C&L)  Report  generation  due  to 
OSS&E  links  with  HSI. 
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7.5.2 

Final  Reports 

Reports  must  address  each  of 
the  COIs,  the  system’s 
operational  effectiveness, 
suitability,  additional  information 
on  operational  capabilities,  and 
include  an  assessment  of 
operational  mission  impacts. 

Reports  must  address  each  of 
the  COIs,  the  system’s 
operational  effectiveness, 
suitability,  safety,  usability, 
additional  information  on 
operational  capabilities,  and 
include  an  assessment  of 
operational  mission  impacts. 

Safety  is  an  HSI  cost  driver. 
Usability  is  a  specific  attribute 
identified  by  test  community  as  a 
priority  for  evaluation. 

7.5.6 

Capabilities 

and 

Limitations 

Reports 

...The  level  of  detail  provided 
will  vary  depending  on  the 
amount  of  preexisting 
information  available,  the 
warfighter’s  need  for  technical 
information,  and  the  amount  of 
time  and  resources  available  to 
conduct  additional  testing  before 
the  fielding  decision.  The  C&L 
Report  should  not  make  specific 
recommendations  concerning 
the  system  fielding  decision  or 
release  for  training  purposes... 

...The  level  of  detail  provided 
will  vary  depending  on  the 
amount  of  preexisting 
information  available,  the 
warfighter’s  need  for  technical 
information,  and  the  amount  of 
time  and  resources  available  to 
conduct  additional  testing  before 
the  fielding  decision.  The  C&L 
Report  must  include  human- 
related  findings,  limitations,  and 
conclusions.  The  C&L  Report 
should  not  make  specific 
recommendations  concerning 
the  system  fielding  decision  or 
release  for  training  purposes... 

7.6 

Operational 
Test  Report 
Distribution 

SAF/AQ  AFHSIO  should  be 
included  in  the  distribution. 

7.9 

Briefing  Trail 

SAF/AQ  AFHSIO  should  be 
included  in  test  report  briefings. 

8.4.2 

Early 

Influence 

Team 

HSI  should  be  represented 
among  the  Early  Influence  Team 
subject  matter  experts  to  help 
author  T&E  measures,  criteria 
and  scenarios. 
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8.4.3 

Integrated 

Test  Team 
Formation 

The  ITT  should  include  HSI 
representation. 

8.5.3 

Integrated 

Test  Concept 
and  TEMP 
Development 

HSI  should  be  represented  on 

ITT  to  support  scenario,  COI, 

CTP,  MOE  and  MOS 
development. 

8.7.1 

Integrated 

T&E  Support 
to  Consent  to 
Ship  and 
Launch 
Approval 
Decisions 

HSI  analysts  should  participate  in 
OT&E  Phase  II  testing  to  assess 
and  evaluate  the  operations 
segment. 

Atch  1 

References 

" 

Air  Force  Human  Systems 
Integration  Handbook 

" 

Atch  1 

Terms 

Human  Systems  Integration 
(HSI)--A  disciplined,  unified  and 
interactive  systems  engineering 
approach  to  integrate  human 
considerations  into  system 
development,  design,  and  life 
cycle  management  to  improve 
total  system  performance  and 
reduce  costs  of  ownership. 
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Appendix  E:  Recommendations  to  AFMAN  63-119,  Certification  of  System 
Readiness  for  Dedicated  Operational  Testing 

This  appendix  contains  recommendations  for  modifications  to  AFMAN  63-1 1 9.  Recommended  additions  or  modifications  are  in  red. 
Because  the  study  sought  interaction  points  between  the  HSI  and  T&E  processes,  the  “HSI  Touch  Point”  column  is  included  to 
indicate  HSI  activities  associated  with  those  described  in  the  AFI  that  would  improve  HSI’s  treatment  in  T&E  or,  more  generally,  in 
acquisition.  Red  indicates  additions  to  existing  text. 
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1 .4.2.1 

Ensure  a  robust  systems 
engineering  process  is  the 
underlying  foundation  for 
systems  development  and  for 
reviewing  these  templates. 

Ensure  a  robust  systems 
engineering  process  that 
includes  human  systems 
integration  is  the  underlying 
foundation  for  systems 
development  and  for  reviewing 
these  templates. 

HSI  processes  are  integral  to 
best  systems  engineering 
practice. 

1. 4.2.2 

Ensure  their  system  is  mature 
and  demonstrates  stabilized 
performance  in  an  operationally 
relevant  environment  prior  to 
certification.  Additionally,  all 
necessary  test  support  must  be 
available  and  the  system  must 
have  a  high  likelihood  of  a 
successful  operational  test. 

Ensure  their  system  is  mature 
and  demonstrates  stabilized 
performance  in  an  operationally 
relevant  environment  prior  to 
certification.  Operational 
relevance  includes  consideration 
of  the  target  audience  who  will 
operate,  sustain  and  threaten 
the  system.  Additionally,  all 
necessary  test  support  must  be 
available  and  the  system  must 
have  a  high  likelihood  of  a 
successful  operational  test. 

The  operational  environment 
includes  the  blue  and  red  forces 
that  encounter  the  system. 

2.6.1 

Pre- 

Certification 

Reviews 

HSI  SMEs  need  to  be  review 
participants. 
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2.6.1. 1 

" 

- 

" 

HSI  SMEs  need  to  be  part  of  the 
assessment. 

2. 6. 2. 2 

Develop  Exit 
Criteria 

“ 

" 

HSI  should  participate  in  defining 
exit  criteria. 

2.7.1.7 

Areas  where  HSI  disagrees  with 
the  certification  should  be 
included. 

2.7.1.8 

Specifically  provide  details  of 

HSI  readiness,  including  system 
limitations  due  to  HSI  issues  and 
system  attributes  not  testable 
due  to  HSI  issues. 

3.3 

Answering 
Template  Line 
Items 

The  word  “system”  refers  to 
software  as  well  as  hardware 
components  of  the  program 
under  review. 

The  word  “system”  refers  to 
software  as  well  as  hardware 
and  human  elements 
components  of  the  program 
under  review. 

Atch  1 

References 

" 

Air  Force  Human  Systems 
Integration  Handbook 

Abbreviations 
and  Acronyms 

" 

HSI  -  Human  Systems 

Integration 
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Terms 

Human  Systems  Integration 
(HSI)--A  disciplined,  unified  and 
interactive  systems  engineering 
approach  to  integrate  human 
considerations  into  system 
development,  design,  and  life 
cycle  management  to  improve 
total  system  performance  and 
reduce  costs  of  ownership. 

Operational  Concept  -  A  plan  for 
day-to-day  operations  that 
describes  capability 
characteristics  and  goals,  and 
explains  the  how  the  capability 
will  be  employed  by  defined 
operators  to  achieve  operational 
objectives. 

A2.2.1 

Ensure  significant  HSI-related 
events  (e.g.,  approval  of 
Environmental  Impact 

Statement,  training  events  etc.) 
are  included  in  the  schedules 
(PM). 

A3. 2.1 

All  relevant  costs  must  be 
identified,  preferably  using 
objective  engineering  and 
business  estimates  using 
accepted  Air  Force  cost  analysis 
principles  and  processes.  (PM) 

All  relevant  costs  must  be 
identified,  preferably  using 
objective  engineering,  an4 
business,  and  human  resource 
estimates  using  accepted  Air 
Force  cost  analysis  principles 
and  processes.  (PM) 

Manpower,  personnel  and 
training  costs  must  be  included  in 
cost  estimates. 
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A3. 2. 2.1 

Ensure  that  assumptions  and 
constraints  regarding  manpower, 
personnel,  and  training  are 
explicitly  identified.  (User) 

Include  specific  goals  to 
eliminate  manpower 
authorizations,  combine  Air 

Force  Specialty  Codes  (AFSCs), 
or  merge  training  programs. 

A3.2.2.2 

Ensure  that  HSI-specific  issues 
from  the  existing  capability  are 
explicitly  identified  so  they  are 
included  in  the  trade  space. 

(User) 

A3. 2. 3 

Acceptable  ranges  of 
performance  must  be 
established  using  rigorous  cost- 
benefit,  trade-off,  and  sensitivity 
analyses  to  show  decision 
makers  when  and  where  certain 
degradations  in  system  cost  or 
performance  yield  outcomes  that 
no  longer  satisfy  the  mission 
need.  (User) 

Acceptable  ranges  of 
performance  must  be 
established  using  rigorous  cost- 
benefit,  trade-off,  and  sensitivity 
analyses  to  show  decision 
makers  when  and  where  certain 
degradations  in  system  cost  or 
performance  yield  outcomes  that 
no  longer  satisfy  the  mission 
need.  Ensure  consideration  for 
humans  is  factored  into  the 
overall  trade  space.  (User) 

A3. 2.4 

All  nine  HSI  domains  should  be 
explicitly  addressed  in  the  AoA, 
and  potential  issues  clearly 
identified  (User). 

A3. 3 

Refer  to  T&E  Framework  HSI 
examples  for  recommendations 
on  how  to  develop  HSI-related 
MOEs. 

A4.1 

Make  sure  HSI-related 
requirements  are  captured  in  the 
CBRD. 
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A4.8 

Refer  to  T&E  Framework  HSI 
examples  for  recommendations 
on  how  to  develop  HSI-related 
MOEs. 

A4.10 

Refer  to  T&E  Framework  HSI 
examples  for  recommendation  on 
how  to  develop  HSI-related 

KPPs,  KSAs,  and  APAs. 

A4.10.1 

Exemptions  to  mandatory  KPPs 
must  be  explicitly  stated.  (User) 

All  mandatory  KPPs  directly  or 
indirectly  influence  HSI 
requirements. 

A4.10.2 

An  explanation  for  why  the 

System  Training  KPP  does  not 
apply  must  be  given  (User). 

Training  is  one  of  the  nine  HSI 
domains. 

A5.4.5 

" 

“ 

Threats  to  humans  must  be 
explicitly  identified  (PM). 

A6.2.4 

The  ITT  shall  include  a  specific 
individual  identified  as  an  HSI 
subject  matter  expert  who  will 
have  “reachback”  to  all  HSI 
domains.  (PM) 

A6.3.1 

For  programs  with  high  HSI 
interest,  it  is  highly 
recommended  to  form  an  HSI 
subgroup  to  ensure  proper 
attention  to  human-related  T&E. 
(ITT) 

A6.5.1 

Incorporate  standards  for  human- 
related  measurements  (formulas, 
definitions,  and  terms). 
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A6.6 

Common  means  for  successfully 
determining  results  of  testing 

HSI  requirements  are  clearly 
documented  and  approved. 

(ITT) 

A7.2 

(M&S,  interoperability 
certification,  Information 
Assurance  testing,  LFT&E, 
contractor  testing,  etc.) 

(M&S,  human-in-the-loop, 
interoperability  certification, 
Information  Assurance  testing, 
LFT&E,  contractor  testing,  etc.) 

A7.4.1 

HSI  risks  and  mitigation 
methods  and  techniques  are 
clearly  identified.  (PM) 

A7.6 

Refer  to  T&E  Framework  HSI 
examples  for  recommendation  on 
how  to  develop  HSI-related 

MOEs,  MOSs,  COI,  and  CTPs. 

A8.1.2 

Human  interactions  must  be 
clearly  identified  within  all 
concepts,  strategies,  methods, 
and  tactics.  (User) 

A8.2 

" 

“ 

Human  elements  contribute  to 
operational  effectiveness. 

A9.1.2 

(such  as  long  lead  items,  TOs, 
support  equipment,  training) 

(such  as  long  lead  items,  TOs, 
support  equipment,  training, 
workload  and  personnel  skills 
and  knowledge  shortfalls) 

A9.4 

HSI  should  be  involved  in  task 
analysis  in  support  of  suitability 
test  scenario  development. 

A9.7 

HSI  should  engage  in  DOTMLPF 
assessment  due  to  HSI  domain 
alignment  with  DOTMLPF 
constituents. 
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A9.9 

HSI  needs  to  be  part  of  the 
reliability  and  maintainability 
growth  plan  and  its  coordination. 

A10.1 

- 

" 

- 

Human  information  requirements 
flow  from  the  NR-KPP. 

A10.1.1.1 

Ensure  architecture  views  that 
call  out  humans  are  clearly 
detailed.  (User. 

A10.1.2 

Human-to-human  and  human-to- 
machine  interfaces  are  key 
interfaces. 

A10.1.3 

Human  aspects  of  Information 
Assurance  include  interactions  of 
facilities  layouts  and  displays. 

A10.2 

HSI  scope  extends  to  nine  of  the 
thirteen  steps  required  for 
Information  Support  Plan 
preparation. 

All. 4.3 

" 

- 

" 

HSI  M&S  assets  should  be 
included. 

All. 4.4 

“ 

- 

“ 

HSI  M&S  assets  must  be 
verified. 

All. 5.1 

Critical  issues  that  must  be 
addressed  in  OT&E  must  include 
human-related  ones. 

A12.2 

HSI  should  review  definitions, 
formulas  and  evaluation  criteria 
in  all  test  plans  and  T&E 
documents  to  ensure  human- 
element  contributions  to  OSS&E 
are  consistently  and  completely 
represented. 

A12.4.1 

- 

" 

- 

HSI  must  provide  required  inputs 
in  agreed-upon  forms. 
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A12.4.2 

Human  element  test 
configurations  must  be  also  be 
controlled  by  HSI  in  order  to 
contribute  to  reliability  curve 
improvement. 

A13.2.5 

Ensure  HSI  implications  have 
been  addressed  in  the  OPSEC 
plan.  (PM) 

A14.2.1 

Inspection  of  HSI  requirements 
flow  down  should  be  included  in 
formal  reviews. 

A14.2.8 

Ensure  the  contractor’s  test 
program  gives  full  attention  to 
verification  and  validation  of 
human-related  requirements. 

(PM) 

A14.3.1 

HSI  is  an  essential  component  of 
systems  engineering  and  should 
participate  in  test  results 
analysis. 

A14.6.3 

HSI  should  participate  in 
decisions  to  modify  pass/fail 
values  and  document  rationales 
for  elections. 

A15.2.1 

Ensure  that  any  trade-offs  where 
human-related  requirements  are 
significant  are  reported  and  user 
concurrence  is  obtained.  (PM) 
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A15.3.3 

The  system  must  demonstrate 
the  capability  to  satisfy  each  of 
the  14  elements  of  operational 
suitability  and  the  ten  logistics 
support  elements.  (PM)  (See  A9, 
A25,  A27,  A30,  A32,  A33) 

The  system  must  demonstrate 
the  capability  to  satisfy  each  of 
the  14  elements  of  operational 
suitability,  the  nine  HSI  domains, 
and  the  ten  logistics  support 
elements.  (PM)  (See  A9,  A25, 
A27,  A30,  A32,  A33,  A34) 

A15.6.2 

HSI  must  ensure  environment  is 
operationally  relevant  from  the 
viewpoint  of  the  system 
operators. 

A15.10.1 

If  there  are  any  incomplete  test 
areas,  explain  why  and  give 
impacts  on  dedicated  OT&E. 
(RTO) 

If  there  are  any  incomplete  test 
areas,  explain  why  and  give 
impacts  on  dedicated  OT&E. 
Ensure  that  HSI-related  impacts 
are  clearly  identified.  (RTO) 

A16.1.1 

Human  related  aspects  of 
interoperability  (standardization 
of  interfaces,  safety,  task 
interfaces)  must  be  included  in 
system-of  systems  test. 

A16.2 

- 

- 

- 

Exit  criteria  must  include  usability 
and  resilience  studies. 

A16.3 

HSI  should  participate  in  tracing 
user  interface  and  safety 
requirements  as  part  of  the 
measurement  process. 
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A16.5.1 

The  software  must  be  analyzed 
for  safety  critical  functions  and 
determined  acceptable  for 
operational  use.  (PM) 

The  software  must  be  analyzed 
for  safety  critical  functions  and 
determined  acceptable  for 
operational  use.  The  analysis 
shall  consider  the  impact  of 
multiple  minor  issues 
accumulating  to  become  a  major 
safety  issue.  (PM) 

HSI  should  participate  in  safety 
critical  analyses. 

A17.9 

Ensure  that  human  survivability 
has  been  fully  addressed 
throughout  the  LFT&E  process. 
(PM) 

HSI  must  make  sure  human 
survivability  is  in  the  test  plan 
and  analyze  related  test  results. 

Atch  18 

Modeling  and 

Simulation 

(M&S) 

A18.1 

- 

- 

- 

- 

A18.2 

" 

- 

HSI  M&S  requirements  should  be 
incorporated  into  the  MSSP. 

A18.2.4 

- 

- 

Ensure  that  humans  are  properly 
modeled  throughout.  (PM) 

- 

A18.2.5 

Ensure  that  validated  HSI 
models  (e.g.,  Jack,  IMPRINT) 
are  being  used  for  human 
related  M&S.  (PM) 

A18.3 

HSI  should  develop  training  for 
the  M&S  that  will  be  used  for 

T&E. 

A18.4 

Identify  HSI  M&S  tools  that  have 
already  been  validated.  Define 
how  other  HSI  M&S  tools  will  be 
verified  and  validated. 

A18.4.1 

" 

- 

" 

HSI  methods  should  be  applied 
to  scenario  development. 
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18.4.4 

Ensure  that  the  results  of 
human-based  M&S  are  reviewed 
by  appropriate  subject  matter 
experts.  (PM) 

A19.1.1 

e.g.,  hardware,  software, 
support  equipment,  spares,  GFE 

e.g.,  hardware,  software, 
human,  support  equipment, 
spares,  GFE 

A19.2.3 

Ensure  that  requirements  for  test 
subjects  are  defined  within  the 
parameters  of  all  of  the 
capability’s  human-related 
requirements  and  thoroughly 
documented.  (PM) 

A19.3.1 

HSI  needs  to  participate  in  the 
determination  that  form,  fit  and 
function  are  not  adversely 
affected. 

A19.3.2 

- 

" 

- 

HSI  must  execute  trade-off 
studies  to  support  assessment. 

A20.4.2 

Ensure  that  a  qualified  HSI 
subject  matter  expert  is 
assigned  to  the  DR  Review 

Board  (PM). 

A20.5.2 

HSI  needs  to  participate  in  the 
determination  that  form,  fit  and 
function  are  not  adversely 
affected. 

A20.5.3 

HSI  techniques  should  be  used 
to  connect  the  operator  and 
sustainer  to  DRs  and  impacts. 
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A20.8 

HSI  needs  to  be  represented  on 
the  Joint  Reliability  and 
Maintainability  Evaluation  Team 
and  the  Board  to  review  human- 
element  impacts  to  Reliability, 
Maintainability  and  Availability. 

A21.1 

Test  articles  (to  include  support 
equipment,  software,  GFE)  must 
be  as  production-representative 
as  possible  to  support  the 
dedicated  OT&E  and  schedule. 

Test  articles  (to  include  support 
equipment,  software,  GFE,  and 
human  elements)  must  be  as 
production-representative  as 
possible  to  support  the 
dedicated  OT&E  and  schedule. 

A21.2.1 

HSI  issues  must  be  clearly 
documented  to  ensure  there  are 
no  limitations  imposed  on 
production  representative 
articles  that  may  drastically  alter 
OT&E  (PM). 

A22.1 

HSI  is  integral  to  operational 
effectiveness  and  suitability  and 
its  products  should  help  to  define 
relevant  scenarios  and 
environment. 

A22.3 

Testing  related  to  human- 
performance  must  be  defined  in 
order  to  ensure  the  integrated 
system  allows  operators  to 
satisfy  mission  requirements. 

A22.3.1 

Integration  among  system 
components,  subsystems,  and 
external  systems  must  optimize 
total  system  design  and 
performance  capabilities.  (PM) 

Integration  among  system 
components,  subsystems,  and 
external  systems  (including 
humans)  must  optimize  total 
system  design  and  performance 
capabilities.  (PM) 

Suboptimal  solutions  will  result  if 
human  elements  are  not 
considered  in  the  overall 
integration. 
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A22.4 

If  the  system  was  planned  with 
an  evolutionary  acquisition 
strategy,  describe  what 
capabilities  are  lacking  at  this 
time  and  when  they  will  be 
implemented.  (PM) 

If  the  system  was  planned  with 
an  evolutionary  acquisition 
strategy,  describe  what 
capabilities  are  lacking  at  this 
time  and  when  they  will  be 
implemented.  Clearly  identify 
operator  or  maintainer 
capabilities  that  may  be  initially 
lacking  due  to  an  evolutionary 
acquisition  strategy.  (PM) 

A22.5 

Force  protection  issues  need  to 
be  included  in  assessments  of 
survivability  and  lethality 
achievement. 

A23.1.1 

HSI  must  participate  in 
characterization  of  operations 
and  support  environments  from 
the  operator  and  sustainment 
personnel  perspective  in  order  to 
fully  test  operational  suitability. 

To  achieve  this,  HSI  methods 
should  be  employed  as  part  of 
test  scenario  development. 

A23.1.2 

HSI  must  participate  in 
development  of  the  Logistics 
Support  Concept  by  performing 
task  and  workload  analysis. 

A23.3.1 

HSI  methods  should  be 
employed  as  part  of  test  scenario 
development. 

A23.3.2 

Human-related  COIs,  MOEs,  and 
MOSs  must  be  derived  from 
higher-level  sources  by  HSI. 
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A23.3.5 

" 

- 

" 

HSI  should  review  requirements, 
methodologies  and  MOEs. 

A23.3.6 

" 

" 

" 

HSI  M&S  assets  should  be 
documented  in  the  MSSP. 

A23.6 

(e.g.,  lack  of  test  articles,  time, 
system  capabilities,  insufficient 
realism) 

(e.g.,  lack  of  test  articles,  time, 
system  capabilities,  insufficient 
realism,  poor  training  of  human 
subjects) 

A24.8 

Adequate  training  shall  be 
provided  for  all  test 
instrumentation,  data  analysis 
equipment,  M&S  assets, 
associated  test  assets  prior  to 
the  beginning  of  each  test 
phase.  (PM  /  OTA) 

A25.2.2 

A25.2.2.  All  system-related 

ESOH  risks  have  been  accepted 
at  the  appropriate  management 
level  prior  to  exposing  people, 
equipment,  or  the  environment 
to  known  hazards.  A  safety 
release  must  be  provided  to  the 
OTA  before  start  of  dedicated 
operational  testing.  (PM) 

A25.2.2.  All  system-related 

ESOH  risks,  including  those  for 
support  and  packaging,  handling 
and  transportation  equipment, 
have  been  accepted  at  the 
appropriate  management  level 
prior  to  exposing  people, 
equipment,  or  the  environment 
to  known  hazards.  A  safety 
release  must  be  provided  to  the 
OTA  before  start  of  dedicated 
operational  testing.  (PM) 

A26.4 

Document  all  training  that  is 
required  beyond  that  which  is 
necessary  to  fulfill  OT&E.  (PM) 

T raining  for  OT&E  may  be  a  sub¬ 
set  of  the  entire  training 
requirement  for  the  capability. 

A27.6 

Document  all  support  equipment 
that  may  be  required  beyond 
that  which  is  necessary  to  fulfill 
OT&E.  (PM) 

There  may  be  support  equipment 
that  is  not  necessary  for  OT&E 
but  that  is  required  for  the  full 
capability. 
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A30.1.1 

HSI  should  be  involved  in  task 
analysis  in  support  of 
maintenance  test  scenario 
development. 

A30.5 

OT&E  test  team  maintenance 
personnel  must  be  adequately 
trained.  (PM) 

OT&E  test  team  maintenance 
personnel  must  be  adequately 
trained  (including  safety 
training).  (PM) 

A31.1 

HSI  task  analyses  should  be 
used  to  ensure  all  personnel  and 
manpower  requirements  have 
been  identified  for  operations  and 
sustainment. 

A31.1.1 

Document  all  Air  Force 
specialties  that  may  be  required 
beyond  that  which  is  necessary 
to  fulfill  OT&E.  (PM) 

The  specialties  required  for 

OT&E  may  be  a  sub-set  of  that 
which  is  required  for  the  full 
capability. 

A33.1 

Operator  and  maintainer 
technical  data  (i.e. ,  TOs, 
engineering  drawings, 
specifications,  standards, 
process  and  user  manuals, 
technical  reports,  catalog  items) 
are  available  to  support  the 

OT&E  plan  and  schedule.  (PM) 

Operator  and  maintainer 
technical  data  (i.e.,  TOs, 
engineering  drawings, 
specifications,  standards, 
process  and  user  manuals, 
technical  reports,  catalog  items, 
embedded  training,  diagnostics, 
prognostics  and  maintenance) 
are  available  to  support  the 

OT&E  plan  and  schedule.  (PM) 

A33.1.4 

Document  all  technical  data  that 
may  be  required  beyond  that 
which  is  necessary  to  fulfill 

OT&E.  (PM) 

The  technical  data  required  for 
OT&E  may  be  a  sub-set  of  that 
which  is  required  for  the  full 
capability. 

Atch  34 

Human 

Systems 

Integration 
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A34.1 

The  integrated  system  consists 
of  hardware,  software  and 
human  elements.  Require¬ 
ments  related  to  the  human 
elements  of  the  system  must  be 
developmentally  tested  at  the 
system  level  prior  to  starting 
dedicated  OT&E.  (PM) 

Reference  AFI  63-1201. 

A34.1.1 

All  safety  risks  identified  in 
developmental  testing  must  be 
incorporated  into  the  program 
risk  matrix  and  assessed  for  risk 
level.  (PM)  (See  A25) 

A34.2 

Developmental  evaluation  of 
operational  effectiveness  and 
suitability  must  incorporate 
representative  personnel  who 
will  operationally  employ, 
support  and  maintain  the 
system.  (PM)  (See  A15,  A17, 

A25,  A26,  A31 ) 

A34.3 

CBRD  mission-effectiveness 
requirements  must  be 
decomposed  to  include  the 
interactions  of  representative 
personnel  with  the  system  that 
affect  mission  outcomes.  (PM) 
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A34.4 

Operationally  relevant  scenarios 
must  introduce  induced, 
operational  stressors  that  would 
degrade  mission  performance  at 
optempo.  (PM)  (See  A22) 

A34.5 

DT&E  must  test  conditions  for 
usability  (illumination,  vibration, 
sound,  in  MOPP  4  gear;  war, 
peace,  standby  conditions; 
transportation,  access, 
maintenance,  support,  training 
modes)  by  representative 
personnel.  (RTO) 
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Appendix  F:  Air  Force  Test  and  Evaluation  Guidebook  Recommendations 

This  appendix  contains  recommendations  for  modifications  to  the  Air  Force  Test  and  Evaluation  Guidebook.  Recommended 
additions  or  modifications  are  in  red.  Because  the  study  sought  interaction  points  between  the  HSI  and  T&E  processes,  the  “HSI 
Touch  Point”  column  is  included  to  indicate  HSI  activities  associated  with  those  described  in  the  AFI  that  would  improve  HSI’s 
treatment  in  T&E  or,  more  generally,  in  acquisition.  Red  indicates  additions  to  existing  text. 


Section 

Section  Title 

As  Is 
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HSI  Touch  Point 

2.2.2 

Section  2366,  Major 
Systems  and 

Munitions  Programs; 

--  Survivability 

Testing  and  Lethality 
Testing  Required 
Before  Full-Scale 
Production 

[add  new  2.2.2.x  at  end] 

While  not  explicitly  cited  in 

Title  10,  survivability  and 
lethality  testing  implicitly 
requires  specific  testing  to 
ensure  the  survivability  of 
crew  members  and  required 
passengers.  It  is  important 
when  LFT&E  is  in  the 
planning  stages  that  survival 
of  crew  and  passengers  is 
considered. 
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2.2.3 

Section  2399, 
Operational  Test  and 
Evaluation  of 

Defense  Acquisition 
Programs. 

[add  new  2.2.3.x  at  end]  It  is 
critical  that  OT&E  provide  as 
realistic  an  environment  as 
possible  for  exercising  new 
and  upgraded  capabilities. 
Those  environments  that 
cannot  be  completely 
replicated  must  be  simulated 
in  order  to  ensure  that 
operators  and  maintainers 
will  be  able  to  carry  out  all 
capabilities  in  a  fully 
operational  environment  and 
therefore  be  able  to  provide 
an  accurate  assessment  of 
effectiveness  and  suitability. 

3.2.2 

What  Goes  on  OSD 
T&E  Oversight  List. 

[add  to  bulleted  list] 

Extremely  high  level  of 
human-system  interaction 

Adding  this  would  require 
concurrence  from  OSD 

7.2 

ITT  Formation  and 
Management 

...specialists  in  financial 
management,  contracting, 
safety,  logistics,  training, 
software  development,  etc. 

...specialists  in  financial 
management,  contracting, 
safety,  logistics,  training, 
software  development,  HSI, 
etc. 
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Section 

Section  Title 

As  Is 

Recommended 

HSI  Touch  Point 

7.3 

Integrated  Initial  Test 
Design  (IITD) 

...The  ITT  uses  a  systems 
engineering  approach  to 
identify  and  de-conflict  the 
initial  COIs,  CTPs,  test 
objectives,  MOEs,  resources, 
and  schedules.  It  is 
absolutely  essential  that  HSI 
is  considered  throughout  the 
IITD  process  through  the 
development  of  the  ITC.  The 
IITD  process  culminates  in  an 
ITC  that  includes  an  initial 
description  of  test  scenarios, 
test  locations,  exercises,  T&E 
methodologies,  operational 
impact  assessments  and 
issues,  and  projections  for 
future  increments.... 

9.6.7 

...developmental  testers, 
OSD,  requirements 
sponsors,  test  facilities,  and 
other  stakeholders  as 
needed  during  various  test 
program  phases.... 

...developmental  testers, 

OSD,  requirements  sponsors, 
test  facilities,  HSI,  and  other 
stakeholders  as  needed 
during  various  test  program 
phases.... 

12.1 

Test  Capabilities  and 
Facilities 

Additional  facilities  along 
this  path  are  data 
measurement  facilities, 
system  integration 
laboratories,  hardware-in- 
the-loop  facilities,  and 
installed  system  test 
facilities. 

Additional  facilities  along  this 
path  are  data  measurement 
facilities,  system  integration 
laboratories,  human-in-the- 
loop  facilities,  hardware-in- 
the-loop  facilities,  and 
installed  system  test  facilities. 
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Section 

Section  Title 

As  Is 

Recommended 

HSI  Touch  Point 

Figure  12.1 

T&E  Resource 

Needs  Through 
Program 

Development 

Will  need  to  update  this  figure  to 
include  human-in-the-loop 
facilities  if  the  previous  comment 
is  accepted. 

13 

Deficiency  Reporting 

Recommend  an  HSI  review  of 

TO  00-35D-54,  AFI  63-501,  and 
TO  00-5-1 

13.1 

Accurate 
Categorization  of 

DRs 

The  issue  of  how  to  address  the 
generally  subjective  nature  of 

HSI  DRs  needs  to  be  presented. 

14.1 

Testing  in  Support  of 
Rapid  Response 
Process  (RRP) 

Recommend  an  HSI  review  of 

AFI  63-1 14  along  with  the  entire 
rapid  acquisition  process. 

14.2 

Foreign  Material 
Program  (FMP) 

Recommend  an  HSI  review  of 

AFI  99-1 14  (classified  Secret)  for 
differences  from  the  basic  T&E 
processes. 

Attachment 

1 

References 

[add  new]  Air  Force  Human 
Systems  Integration 

Handbook 

Attachment 

1 

Abbreviations  and 
Acronyms 

[add  new]  HSI  -  Human 
Systems  Integration 

Attachment 

1 

Terms 

[add  new]  Human  Systems 
Integration  (HSI)— A 
disciplined,  unified  and 
interactive  systems 
engineering  approach  to 
integrate  human 
considerations  into  system 
development,  design,  and  life 
cycle  management  to 
improve  total  system 
performance  and  reduce 
costs  of  ownership. 
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Section 

Section  Title 

As  Is 

Recommended 

HSI  Touch  Point 

A5.3.2 

The  Executive  Level 

[add  new  to  end]  The  Air 

Force  HSI  Office,  has  an 
advisory  role  to  SAF/AQ  for 

HSI  matters. 

A5.3.3 

The  Working  Group 
Level 

[add  new  at  end]  HSI  subject 
matter  experts  should  be  part 
of  the  working  level  ITT. 

A5.3.4 

Subgroups 

[add  new]  A5.3.4.3.  In  order 
to  ensure  focused  T&E 
attention  on  HSI,  the  ITT 
should  create  an  HSI 
subgroup  for  any  program 
that  has  extensive  HSI 
requirements  and/or  high- 
level  HSI  interest. 
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Appendix  G:  HSI  Test  and  Evaluation  Framework  Examples 


<D 

-Q 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 

Decision 

3 

£ 

Kei 

/  Requirements  and  T&E  Measures 

OAR) 

Supported 

5 

a. 

<D 

■4-* 

(/) 

>» 

CO 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

T 

L. 

Q) 

-Q 

£ 

o 

CO 

c 

o 

KSA: 

Sustainable 

Does  the  long- 
duration  bomber 
system  support 

Number  of  Crew:  Can  two  (2) 

crewmembers  execute  the  full  range 
of  long-duration  bomber  missions 
and  associated  phases  over  a  24- 
hour  period? 

Crew  exhibits  signs  of  fatigue 
per  FAA  Minimum  Flight  Crew, 

AC  23.1523,  Appendix  1. 

Number  of  missions  and 
associated  phases  successfully 

NASA  TLX  Workload 
assessment  of  five  crews 
executing  mission  scenario 
number  one  (1). 

Piece  validation  using  flight 

MS  B 

£ 

3 

Q 

■ 

O) 

c 

o 

_l 

Crew  Workload 

operation  by  a 

two-person 

crew? 

accomplished  in  24-hours  with 
less  than  3%  errors. 

trainer.  Twelve  (12)  crew  teams, 
randomly  paired  for  36  trials,  four 
(4)  trials  per  mission  scenario 
(including  rest-period  scenarios). 
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E 

<D 

■4-* 

(/) 

> 

CO 


Test  Methodologies  / 

Key  Resources 

(M&S,  SIL,  MF,  ISTF.HITL, 

Decision 

Key  Requirements  and  T&E  Measures 

OAR) 

Supported 

0) 

o 

</> 

c 

o 

o 

L- 

o 

+■» 

5 

<D 

a 

O 

E 

Q) 

4-1 

(fl 

>. 

V) 


<D 

< 

■a 

<u 

c 

c 

ns 

E 


Key 

Requirements 
(KPPs,  KSAs) 


KPP: 

Sustainment 

KSA: 

Reduced 
Manpower  Costs 


Critical 
Operational 
Issues  (COIs) 


Can  1UOX  (Air 
Force  Specialty 
Code)  personnel 
operate  a  single 
UAS  in  all 
phases  without  a 
rated  18X  UAS 
pilot? 


Key  MOEs/MOSs 


Can  the  operator  console  supply  the 
piloting  skills,  knowledge  and 
abilities  lacking  in  1UOX? 


Critical  Technical  Parameters 
(CTP)  &  Thresholds 


UAS  manual  control  error: 
from  optimal  trajectory 
(integrated  spherical  error). 


<7% 


1 UOX  / 1 8X  Performance 
Ratio:  Ratio  of  1  UOX  control 
error  performance  to  18X 
control  error  performance  > 
95%. 

Average  number  of  aircraft  lost 
per  1000  missions. 

Constrained  flight  collision 
avoidance  during  operations: 
99%  confidence  of  greater  than 
five-inch  distance  from  any 
point  on  UAS  surface  to 
obstacle;  integrated  over  all 
obstacles  when  in  ISR  mode. 


Operator  console  engineering 
model  with  UAS  simulator.  Eight 
(8)  subjects  (two  novice,  two 
journeymen,  two  expert  1UOX, 
two  18x  journeyman  pilots) 
navigating  through  design 
scenario  urban,  mountain,  rural 
waypoint  courses. 


EMDD  engineering  model  and 
instrumented  UAS  test  model. 
Twelve  (12)  subjects  navigating 
through  AFOTEC  Urban  Flight 
Obstacle  Course  (straight, 
corner,  necked  pathway,  dual 
balcony  and  laundry  line  cases). 
Simultaneous  ISR  collections  of 
three  by  five  inch  (3  in  x  5  in) 
ground  targets  during  obstacle 
passes. 


MS  B 


LRIP 


UNCLASSIFIED 
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Unmanned  Aerial  System  Operator  Console  System 


UNCLASSIFIED 


Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

KPP: 

Sustainment 

KSA: 

Reduced 
Manpower  Costs 

Can  1UOX  (Air 
Force  Specialty 
Code)  personnel 
operate  a  single 
UAS  in  all 
phases  without  a 
rated  18X  UAS 
pilot? 

System  Usability:  Does  the 
operator  console  interface  direct 
user  attention  to  critical  information 
and  activation  functions  during 
Handover,  Enroute,  ISR,  and  Strike 
modes? 

Visual  Attention  Directivity 
Efficiency 

Nominal:  <  two  (2)  second  lag 
between  screen  presentation 
and  visual  lock  on  critical 
display 

Fatigue:  <  four  (4)  second  lag 
between  screen  presentation 
and  visual  lock  on  critical 
display 

Visual  Attention  Failure 

Nominal:  <  2%  visual  lock 
failure 

Fatigue:  <  3%  visual  lock 
failure 

Time-compressed  demonstration 
of  Scenario  2C  screen 
progression.  15  subjects  (five  (5) 
novice,  five  (5)  journeymen,  and 
five  (5)  experts).  Nominal  taken 
during  first  watch  by  personnel 
serving  first  watch.  Fatigue  taken 
during  third  watch  by  personnel 
serving  third  watch. 

Eye-tracking  computer  platform 
including  cameras  and  infrared 
emitters  hidden  behind  small  dark 
windows  at  the  top  and  bottom  of 
the  monitor. 

MS  C 
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Test  Methodologies  / 

0) 

Key  Resources 

-Q 

(M&S,  SIL,  MF,  ISTF.HITL, 

Decision 

3 

E 

Kei 

/  Requirements  and  T&E  Measures 

OAR) 

Supported 

> 

<D 

■4-* 

Key 

Critical 

o 

a: 

</) 

> 

CO 

Requirements 

Operational 

Critical  Technical  Parameters 

(KPPs,  KSAs) 

Issues  (COIs) 

Key  MOEs/MOSs 

(CTP)  &  Thresholds 

Workload:  Does  the  crew  have 

Single  UAS  Workload 

IMPRINT  Pro  simulation  of  pilots 

PDR 

capability  to  perform  the  required 

(individual  crew  members) 

and  sensor  operator  performing 

tasks? 

Gaining  Handover  Workload 

all  mission  functions. 

Does  the  crew  have  enough  spare 
capacity  to  take  on  additional  tasks? 

(single  UAS):  <20% 

DMO  exercise  at  Air  National 

Operational 

Enroute  Workload  (Single 

Guard’s  Distributed  Training 

Test 

Does  the  crew  have  enough  spare 

UAS):  <17.5% 

Operations  Center  (DTOC)  to 

Readiness 

<D 

capacity  to  cope  with  emergency 

validate  simulation.  Measures  to 

o 

situations? 

ISR  Workload  (Single  UAS): 

include: 

c 

<20% 

o 

.  NASA  TLX  Workload 

k. 

o 

Strike  Workload  (Single  UAS): 

assessment 

ro 

<20% 

•  Eye  tracking  and  video 

<D 

Q. 

Can  four  UAS  be 

taping  of  pilot  and  SO 

o 

KPP: 

operated  by  a 

Losing  Handover  (Single  UAS): 

scanning  during  CT 

E 

Sustainment 

crew  consisting 

<20% 

scenarios. 

■4-* 

t/> 

KSA: 

of  two  pilots  and 

•  Graduated  insertion  of  five 

>> 

CO 

Reduced 

one  sensor 

Concurrent  Tasking  (CT) 

stressor  tasks  into  CT 

"ro 

Manpower  Costs 

operator  (SO)? 

Definitions 

scenarios. 

k. 

0) 

1)  Two  UAS  in  Strike;  one 

•  Measures  of  situation 

< 

gaining  Handover,  one 

awareness. 

"D 

0 

Losing  Handover 

•  Number  of  times  actions 

c 

c 

2)  Two  UAS  in  ISR;  two  in 

were  made  for  one  plane  but 

<T5 

E 

Strike 

meant  for  a  different  plane. 

c 

3)  One  UAS  Gaining 

•  Number  of  undo  actions 

Handover,  one  Losing 
Handover,  one  in  Strike, 
one  Enroute 

required  during  simulation. 

Workload  Concurrent  Tasking 
(individuals) 

CT1 :  Pilot  <  85%;  SO  <80% 

CT2:  Pilot<80%;  SO<90% 

CT3:  Pilot<75%;  SO<75% 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

in 

Information  System 

KSA: 

IA  Situation 
Assessment 

Can  system- 
specific  training 
be  eliminated? 

Intuitive  Usability:  Can  the  system 
be  used  effectively  by  an  untrained 
individual? 

Successful  exercise  of  core 
interface  functionality. 

Percentage  of  time  eye  motion 
tracks  to  correct  area  of  screen. 

Percentage  of  incidents  where 
eyes  stop  and  remain  on  correct 
area. 

Numbers  of  induced  keystroke 
errors  and  undo’s:  85% 
reduction  from  the  TD 
prototype. 

Facial  expressions:  Less  than 
one  instance  of  frustration  per 
trial. 

Exercise  of  clickable  prototype 
on  Critical  Mission  Segment 
Scenarios  (CMSS)  1-11. 

Minimum  of  10  subjects  pulled 
from  AFSC  OCYC  apprentice 
pool. 

Eye-tracking  computer  platform 
including  cameras  and  infrared 
emitters  hidden  behind  small 
dark  windows  at  the  top  and 
bottom  of  the  monitor. 

Keystroke  tracking  during  10 
hours  of  operation  over  two 
sessions  for  each  of  10  subjects. 

Webcam  capture  of  user's  facial 
expressions.  Synchronized 
audio  channel  to  capture  subject 
vocalizations. 

All  tests  above  simultaneously 
executed. 

MS  C 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

CO 

Airfield  Operations 

KSA: 

Controlled  Air 
Field  Access 

Are  aircraft 
protected  from 
mobile,  ground- 
based  entities 
when  on  the 
flight  line? 

Wildlife  Exclusion:  Are  aircraft 
protected  from  incursions  of  wild 
animals  onto  taxiways  and  runways? 

Perimeter  barrier  height:  15 
feet. 

Perimeter  barrier  spacing:  No 
less  than  90  feet  from  center 
line  of  taxiway  or  runway. 

Inspection  by  base  civil 
engineers. 

Fielding 

MS  C 

Vehicle  Exclusion:  Are  measures 
in  place  to  prevent  unauthorized 
operators  from  driving  vehicles  onto 
the  flight  line? 

Ignition  Control:  Airfield  vehicle 
ignitions  controlled  and 
activated  with  Common  Access 
Card  (CAC). 

CAC  Driver  Authorization:  CAC 
cards  encoded  with  driver 
authorization  credentials. 

Subsystem  demonstration  using 
engineering  model. 

CASE  testing  of  encoding 
module  and  software  system. 

Laboratory  encoding  verification 
of  eight  (8)  driver  authorization 
codes  using  hand  scanner. 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

Fighter  Aircraft  Crew  System 

KPP: 

Force 

Protection 

Does  the  system 
enable  crew 
members  to 
sustain  operation 
under  rapid  onset 
conditions? 

Can  crew  members  sustain  safe  and 
effective  operations  at  up  to  9  Gs 
rapid  onset? 

Pressure  breathing  for  G  (PBG): 
Provide  a  mean  score  of  at 
least  eighty-five  (85)  seconds  in 
the  weighted  time  and 
acceleration  explores  of  a 
relaxed  rapid  onset  series. 

Maximum  possible  score:  105 
seconds. 

Time  to  experience  100% 
peripheral  (or  50%  central  light 
loss):  15  seconds 

Centrifuge  tests  of  rapid  onset 
series: 

1.  Plus  (+)  six  (6)  G  per 

second  to  plus  (+)  three  (3) 
Gs;  hold  for  fifteen  (15) 
seconds;  two  (2)  minute 
rest  period. 

2-7.  Incremental  increases  of 
plus  (+)  1  Gs  up  to  plus  (+) 

9  Gs. 

Subjects  remain  relaxed  (no  anti- 
G  straining  maneuver). 

Tests  cease  when  subjects 
experience  100%  peripheral  or 
50%  central  light  loss. 

Light  bar  apparatus  with  button 
mounted  on  handlebar  grip. 
Subjects  squeeze  grip,  click  the 
button  on  top  in  response  to  red 
lighting  that  moves  from 
periphery  to  center  of  vision. 
Subject  failure  to  press  button 
once/second  indicates  narrowing 
of  field  of  vision.  Subjects  signal 
test  abort  by  releasing  grip. 

Vital  sign  measures  via  chest, 
neck,  and  left  wrist  electrodes. 

Doppler  sensor  measure  of 
velocity  of  blood  flow  to  brain. 

MS  C 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

00 

Fixed-wing  Aircraft 

KPP: 

Sustainment 

KSA: 

Reliability 

Does  the 
performance  of 
system 
maintenance 
tasks  induce 
system  failures? 

Design  Vulnerability:  Is  the  engine 
pylon  design  vulnerable  to 
maintenance  damage? 

BIT:  Does  built-in  test  (BIT)  notify 
flight  and  maintenance  crews  of 
maintenance  errors  (incomplete  or 
incorrectly  performed  maintenance 
procedures)? 

Human  Error  Probability  (error 
occurrences  /  error 
opportunities):  acceptance  < 

0.01 

Validation  of  error  analysis  using 
subsystem  (Pylon,  Engine,  BIT, 
and  Ignition)  and  integrated 
system  simulations. 

Ecological  analysis  of  legacy 
system  procedure  to  validate 

M&S  error  opportunities. 

Flight  model  test  of  procedure  to 
identify  previously  missed  error 
opportunities  and  catalog  error 
occurrences.  Three 
maintenance  crews  performing 
full  engine  change  out. 

MS  B 

MS  B 

MS  C 

Gi 

Information  System 

KPP: 

Materiel 

Availability 

Does  the 
capability  provide 
an  unambiguous 
indication  of 
instantaneous 
materiel 
availability? 

Availability  Determination:  Can 

novice  and  experienced  users 
rapidly  determine  the  exact  amounts 
of  materiel  available? 

Can  novice  and  experienced  users 
accurately  identify  the  exact 
amounts  of  materiel  available? 

Rapid  determination:  Call  up 
materiel  availability  display 
within  two  (2)  seconds  of  voice 
command. 

Accurate  identification:  Vocally 
communicate  the  counts  of  10 
inventory  items  in  eight  (8) 
seconds  with  99%  accuracy 
over  25  trials. 

Timed,  live-virtual  simulation  of 
four  candidate  displays  with  40 
subjects  using  “Engagement 

5.2B”  scenario. 

Timed,  live  simulation  of  three 
candidate  displays  with  20 
subjects.  Vary  ordering  of  items 
numerically,  alphabetically,  and 
categorically. 

MS  B 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameter 
(CTP)  &  Thresholds 

O 

T— 

Information  system 

KPP: 

Sustainment 

KSA: 

Reliability 

Does  the  Data 
Collection,  Analysis 
and  Corrective 

Action  System 
(DCACAS)  support 
reliability  growth 
throughout 
operations? 

Autonomous  Collection:  Does  the 
DCACAS  autonomously  collect 
success  and  failure  data? 

Subsystem  success 
(completion)  sensor  activations: 
100% 

Subsystem  failure  sensor 
activations:  100% 

Human-in-the-Loop  testing  of 
subsystem  maintenance 
procedures.  One  subject.  Full 
demonstration  of  all  maintenance 
procedures. 

Test 

Readiness 

Review 

MS  C 

MS  B 

User  Prompting:  Does  the 

DCACAS  system  effectively  prompt 
flight,  maintenance  and  logistics 
crews  to  record  success  and  failure 
data? 

Incident  Recording  Ratio 
(recorded/opportunity):  98% 

Time  to  prompt 

acknowledgment:  <  5  seconds 

Simulations  of  flight, 
maintenance  and  logistics  crew 
procedures  on  legacy  system 
with  mountable  PDA-based 
display/prompt  system. 

Videotaped  observations  and 
digital  timing  of  prompt 
acknowledgement. 

Availability  and  Usability:  Are 

failure,  analysis  and  corrective 
action  data  accessible  to  all 
sustainment  and  development 
personnel  in  usable  form? 

Time  to  access:  <  5  seconds 

Time  to  correct  input  errors:  <  8 
seconds 

Post-processing  conversion 
operations  required:  0 

Eye-tracking  computer  platform. 
Webcam/audio  capture  of  user's 
facial  expressions  and 
vocalizations. 

Walkthroughs  of  stakeholder 
post-access  information 
processing. 

Subjective  usability  rating 
questionnaire,  Likert-scale 
evaluations. 

UNCLASSIFIED 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameter 
(CTP)  &  Thresholds 

T— 

T“ 

Maintenance  Information  Support  System 

KSA: 

Accessible 

Procedures 

Are  maintenance 
procedures 
available  at  point 
of  application? 

Availability:  Are  searchable,  just- 
in-time  procedures  accessible  to 
maintainers  at  the  location  where 
maintenance  will  be  performed? 

One-touch,  biometric 
accessible:  Procedures 
delivery  system  accessible  from 
any  unit  via  one-touch, 
biometric  activation. 

Time  to  access  correct 
procedure:  <  5  seconds 

Language  accessibility: 
Procedures  available,  including 
keyword  searches,  in  American 
English,  United  Kingdom 

English,  French,  German, 

Polish  and  Arabic. 

Accessibility  in  Environment: 
All-conditions  interface, 
including  temperature,  lighting, 
pressure,  vibration,  dust  and 
vapor  obscuration. 

CASE  testing  of  alphanumeric 
symbols  with  Latin,  Cyrillic  and 
Arabic  alphabets. 

Usability  tests  prior  to  Coalition 
Warrior  Interoperability 
Demonstration  (CWID) 
Interoperability  Trial  (IT).  Five 
(5)  subjects  from  each 
nationality.  Videotaped  trials, 
questionnaire,  Likert  scale 
acceptance  evaluations. 

Mockup  display  testing. 

Videotape  of  candidate  display 
mockups  with  digital  camera 
perpendicular  to  display  field 
Environments  tested  in  order  of 

•  Temperature  -  environment 
chamber  from  minus  20- 
plus120  degrees  Fahrenheit 

•  Lighting  -  bench  test,  dark  to 
full-sun,  fluorescent  and 
incandescent  white,  blue,  red, 
and  green,  hemisphere,  1 .5  ft 
radius. 

•  Pressure  -  environment 
chamber  to  two  atmospheres 

•  Vibration  -  shaker  table  , 

-0.5  dBg,  2300  RPM 

•  Dust  -  Duration  exposure  at 
Yuma  Test  Center 

•  Humidity  -  Condensing 
humidity  tests. 

Test 

Readiness 

Review 

Post 

Deployment 

Review 

MS  B 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

CM 

T— 

Manned  Aircraft  Crew  System 

KSA: 

Personnel 

Survivability 

Can  the  crew 
expeditiously  exit 
the  vehicle? 

Tools:  Do  support  tools  impede 
rapid  evacuation  of  the  vehicle? 

Maximum  time  for  crew  exit: 
ten  (10)  seconds 

Tool  access:  Within  arm  length 
reach  of  personnel  near  the 
boundary  conditions  of  the 
central  95%  of  the 
representative  user  population 
(MIL  STD  1472). 

Timed  walk  though  of  emergency 
egress  tasks. 

Anthropometric  model  of  full- 
scale  mockup  of  all  crew 
stations. 

Inspection  of  full-scale  mockup  of 
single  seat  using  personnel  near 
the  boundary  conditions  of  the 
central  90%  (5th  to  95th 
percentile)  of  the  representative 
user  population. 

MS  C 

UNCLASSIFIED 
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UNCLASSIFIED 


ro 

Dimensions:  Does  the  system 
accommodate  linear  spatial 
dimensions  of  central  95%  of  the 
representative  user  population,  two 
carry-on  items  and  two  stowed 
cargo  items? 

Seat  pitch:  37  inches 

Seat  width:  21  inches 

Seated  headroom:  37  inches 
Carry  on  one  dimensions: 

21 .5  in  x  9  in  x  15.5  in 

Carry  on  two  dimensions: 

16.5  in  x  6  in  x  13  in 

Anthropometric  model  measured 
against  bounding  and  mixed- 
complement  scenarios. 

Have  representative  passenger 
pack  for  actual  mission  and 
attempt  to  stow  cargo. 

MS  B 

o 

k. 

< 

r 

Stowed  cargo  (each  of  2):  62 
linear  inches  (1  +  w  +  h) 

CO 

T“ 

o 

Q. 

</> 

C 

2 

H 

i_ 

0) 

O) 

c 

0) 

KSA: 

Passenger 

Load  and 
Density 

Does  the  capability 
accommodate  the 
required  number  of 
inhabitants? 

Are  stowed  carry-ons  within  the  non- 
extended  reach  of  a  5th  percentile 
female  when  motion  restraints  are  in 
place? 

Non-extended  reach  to  carry- 
ons:  27.6  in. 

Demonstration  of  full-scale 
mockup  of  compartment  section. 
Have  representative  passenger 
pack  for  actual  mission  and 
attempt  to  stow  cargo. 

MS  C 

(/) 

0) 

ro 

0- 

Does  the  capability  support  the 
volumetric  spatial  dimensions  of 
central  95%  of  the  representative 
user  population,  carry-on  items  and 
stowed  cargo  items? 

Small  female  plus 
carry-ons:  29,616.5  cu  in. 

Large  male  plus 
carry-ons:  33,035.2  cu  in. 

Stowed  cargo:  510.8  cu  ft 

Lickert  scale,  user  rating  of 
space. 

Inspection  of  full-scale  mockup  of 
half-row. 

Lickert  scale,  user  rating  of 
space. 

MS  C 

Row  Number 

System 

Ke> 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF,HITL, 
OAR) 

Decision 

Supported 

m 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 

Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

UNCLASSIFIED 
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UNCLASSIFIED 


Personal  Amenities:  Does  the 
system  incorporate  handicap- 
accessible,  private  lavatory  facilities 
in  sufficient  numbers  for  the 
system's  full  50-inhabitant 
complement? 

Max  wait  time:  5  minutes 

Avg.  wait  time:  <  2  minutes 
Handicap  stall  width:  39  in. 
Handicap  stall  depth:  59  in. 

Task  network  simulation 

Interior  CAD  model  of  stall 
employing  "Jack"  anthropometric 
model. 

MS  B 

03 

i- 

Does  the  system  supply  personal 
video  with  audio  to  each  seated 
inhabitant? 

Controls  reachable  from  vertical 
line  from  shoulder  center  to  top 
of  hip  bone:  Large  male  (24  in) 
to  small  female  (27.6  in). 

CAD  models  of  single  seat 
configuration  employing  "Jack" 
anthropometric  models. 

T— 

k. 

< 

r 

o 

Q. 

</> 

C 

2 

H 

k. 

<D 

O) 

C 

<D 

tn 

ro 

KSA: 

Passenger 

Load  and 
Density 

Does  the  capability 
accommodate  the 
required  number  of 
inhabitants? 

Eye-screen  distance:  between 

45  inches  away  at  horizontal 
eye  level  and  35  inches  away  at 
a  30-degree  downward  gaze 
angle.  It  shall  not  be  closer 
than  the  resting  point  of 
vergence  for  any  passenger  in 
the  small  female  through  large 
male  population. 

CAD  models  of  single  seat 
configuration  employing  "Jack" 
anthropometric  models. 

Q_ 

Visibility  in  full  sun:  ability  to 
discriminate  all  colors  and 
contrasts. 

1 0  user  trials  of  single-seat,  full- 
scale,  mockup  of  color  bars  with 
grey  scale  and  plunge  patterns  in 
hemispherical  screen- 
impingement  pattern  of 
simulated,  full  sunlight. 

Eight-hour,  full  scale  trial  of 
system  with  full  50-person 
complement. 

MS  C 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

T— 

Training  Information  System 

KPP: 

System  Training 
KSA:  Coordinate 
Civilian 

Personnel  and 
Contractor 

Support 

Does  the 
capability  meet 
prior-to-forward- 
deployment 
training 
objectives  for 
civilians  and 
contractors? 

Training  Availability:  Does  the 
capability  support  seamless  civilian 
and  contractor  training  at  CONUS 
locations,  when  in  transit  and  at  the 
staging  location? 

User  access:  Secure  access  to 
login,  course  materials  and 
trainer  feedback  from  a)  all 
Government  Education  and 
Training  Network  (GETN)  sites; 
b)  air,  sea  and  ground  transport 
systems;  and  in-theater 
housing. 

Progress  handoff:  Resume 
active  training  module  from 
exact  point  of  last  trainee-to- 
training-system  interaction  with 
no  more  than  a  five-second 
handoff  lag. 

Communication  Audibility: 

Voice  channel  between  trainers 
and  trainees  audible  when 
trainee  is  in  85  dB  noise  floor 
environment. 

Distortion  in  voice  channel 
<.0025%  from  20  to  20kHz. 

Assessment  at  25  distributed 
sites  using  modified  GETN 
automated  software  test  cases. 
Three  shifts  tested  on  eight-hour 
centers  from  all  test  sites. 

White  event  C2-ground 
demonstration  of  capability  at  Air 
National  Guard’s  Distributed 
Training  Operations  Center. 

Noise  measurements  taken  from 
20Hz  to  20kHz  in  10  Hz  intervals 
in  anechoic  chamber.  On-line 
computer-controlled  acquisition 
and  processing. 

Confirmation  of  anechoic  findings 
using  Modified  Rhyme  Tests  of 

20  human  subjects  in  presence 
of  increasing  noise  floor  starting 
from  measured  ambient  to 
induced  85  dB. 

Measurements  of  transmission  to 
ear-piece,  end-to-end  signal 
using  spectrum  analyzer. 

Service/ 

JROC 

Validation 

and 

Approval 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

CD 

T— 

Training  Information 
System 

KPP: 

System  Training 
KSA: 

Coordinate 

Civilian 

Personnel  and 
Contractor 

Support 

Does  the 
capability  meet 
prior-to-forward- 
deployment 
training 
objectives  for 
civilians  and 
contractors? 

Automated  Training  Assessment: 

Does  the  capability  accurately 
assess  civilian  and  contractor 
preparedness  for  forward 
deployment? 

Training  Module  Assessment 
vs.  Simulation  Score:  Pearson 
correlation  coefficient  (r)  of  90, 
with  95%  regression 
confidence. 

Pre-training  Full  Spectrum 

Civilian  simulation  scoring. 

Embedded  training  assessment 

Post-training  Full  Spectrum 

Civilian  simulation  scoring. 

Fifteen  subjects.  Statistical  data 
reduction. 

Service/ 

JROC 

Validation 

and 

Approval 

1"- 

T— 

Unmanned  Aerial  System  Ground  Control  System 

KSA: 

Positive 

Indication 

Does  the  Ground 
Control  System 
(GCS)  provide 
Positive 

Indication  of  UAS 
states  when 
transferred  in? 

Unacknowledged  Commands: 

Does  the  GCS  provide  positive 
indication  that  unacknowledged 
commands  from  previously 
controlled  UAS’s  have  been  cleared 
from  the  system  prior  to  transfer  in? 

GCS  command  buffer  flush: 
Within  25  ms  of  receipt  of  CTA 
(Command  Transfer 
Acknowledgment)  telemetry 
signal  from  UAS. 

CASE  test  of  algorithm. 

Built-in,  timed  test  of  full  flush. 

GCS  test  equipment  to  include 
command  buffer  stored  volume. 

MS  C 

MS  B 

MS  C 

Autopilot  Engagement:  Does  the 
GCS  provide  continuous  positive 
indication  when  autopilot  is 
engaged? 

Autopilot  engagement  display: 
All-conditions  display  within 
one-inch  radius  of  autopilot 
switch. 

Autopilot  engagement  display: 
Line-of-sight  visibility  to  seated 
pilot,  all  chair  adjustments. 

Inspection  under  all  lighting 
conditions. 

JACK  ergonomic  model  5th 
percentile  female  to  95th 
percentile  male. 

External  Pilot  Point  of  View:  Does 
the  EP  controller  provide  positive 
indication  of  direction  (forward,  rear) 
when  associating  right-left  controls 
with  actuators? 

EP  POV  Indicator:  Zero-error 
left-flap,  right-flap  activation  in 
all  relative  orientations. 

Live-Virtual  exercise  with  UAS 
trailer  and  engineering  model  of 

EP  controller,  randomized  15 
deg  increments  for  full  360  deg. 
horizontal  coverage. 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

00 

T— 

Unmanned  Aerial  System  Maintenance  Training  System 

KSA 

Rapid  Sortie 
Turnaround 

Does  the 
capability  support 
rapid  training  of 
users? 

Amount  of  training  time  required: 

Does  system-specific  training 
achieve  required  competence  in 
accordance  training  plan  and 
operational  need? 

Number  of  repetitions 
required  to  satisfactorily 
achieve  competence  scores: 

<  3  repetitions  per  embedded 
module. 

Required  Training  Time:  <  1 

hour  of  training  spent  on  each 
embedded  module. 

Maintainers’  reactions  to 
training:  >  4.85  mean  overall 
score. 

Learning  as  a  result  of 
training:  98%  mission  success 
rates;  untimed. 

Results:  50%  decrease  in 
mean  UAS  return-to-service 
times  at  three,  six  and  nine 
weeks. 

Automated  recording  of  trials  by 
maintainers  through  nine  weeks 
from  IOC. 

Automated  recording  of  time 
maintainers  are  logged  into 
specific  training  module  through 
nine  weeks  from  IOC. 

Lickert  scale,  self-evaluation  of 
training  relevance  to 
maintenance  tasks. 

Increases  in  success  rates  on 
training  scenarios  for  critical 
mission  segment  4,  5  and  1 1  at 
three,  six  and  nine  weeks. 

Ecological  observation  of 
maintainers.  Samplings  taken  at 
junctures  of  design-reference 
task-network  model,  all  shifts,  ten 
hours’  of  operation  for  each  shift. 

Time  measures  beginning  with 
handoff  at  model’s  upstream 
juncture  through  downstream 
juncture. 

Test 

Readiness 

Review 

UNCLASSIFIED 


Vehicle  Crew  System  System 


UNCLASSIFIED 


Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

KSA: 

Environ¬ 

mental 

Control 

Is  the  environment 
suitable  for  extended 
periods  of  human 
habitation  in  all 
modes  of  operation? 

Level  of  Humidity:  Does  the  level 
of  humidity  in  the  work  environment 
remain  at  50  +/-  20%  when  the 
capability  is  in  operation? 

Humidity  level:  50  +/-  20% 

Ambient  (internal)  humidity  levels 
with  calibrated  electronic 
hygrometer. 

MS  C 

Operational  Temperature:  Is  the 

temperature  of  the  work  area 
maintained  in  the  range  of  18-26°C 
(64-78°F)  when  external  ambient 
temperature  is  between  -20°F  with 
minimum  solar  load  and  +1 1 5°F  with 
maximum  solar  load? 

Workspace  ambient 
temperature:  71  +/-7°F 

External  ambient  temperature: 
-25°F  to  +140°F. 

Dry  bulb  measures  of  internal 
ambient  temperature. 

Stevenson  screen  measurements 
of  external  ambient  temperature. 

MS  C 

Crew  Core  Temperature:  Does  the 
core  body  temperature  of  work  area 
inhabitants  remain  between  95°F 
and  100°F  when  the  capability  is  in 
operation? 

Inhabitant  core  body 
temperature  (rectal):  97.6  +/  - 
2°F 

Rectal  temperature  measurement 
with  probe. 

MS  C 

All  measures  taken  with 
representative  crew  members 
performing  scenario  7  mission 
tasks. 

Key  Resources:  Workspace 
mockup;  thermal  chamber 

UNCLASSIFIED 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameter 
(CTP)  &  Thresholds 

O 

CM 

Aircrew  Kneeboard  Laptop 

KSA: 

Personnel 

Survivability 

Can  emergency 
egress  occur  via 
the  ejection  seat 
with  a  kneeboard 
laptop  system  in 
place? 

Obstruction:  Does  the  system 
obstruct  the  pathway  for  emergency 
egress? 

Ejection  position  with 
kneeboard  laptop  system:  Any 
position  within  5  inches  of 
vertical  adjustment  (MIL-DTL- 
9479E). 

Accommodation  of  flight 
clothing  and  personal 
equipment:  Seat  Assembly  with 
laptop  kneeboard  laptop 
operation  without  snag,  jam  or 
damage  to  clothing  or 
equipment. 

Knee  Room:  No  less  than  25 
inches  high  and  20  inches  wide 

Kneeboard  laptop  cannot 
interfere  with  ejection  rails  at 
any  point. 

Task  modeling  and  simulation  to 
determine  likelihood  of  hung 
jumper 

Monte  Carlo  Seat  Assembly 
simulations  with  Draping  and  Aux 
Equip  model  engaged. 

Height  measurement  of  crew 
compartment  from  interior  floor  at 
seat  edge  and  at  six  inches 
forward  of  seat  edge.  Width 
measurements  extending  from 
seat  edge  to  left  and  right  panels 
perpendicular  to  floor  at  seat 
edge  and  at  six  inches  forward  of 
seat  edge. 

Engineering  model  test  of  Seat 
Assembly  with  kneeboard 
mockup  using  representative 
crash  dummies. 

MS  B 

MS  B 

MS  C 

LRIP 

UNCLASSIFIED 
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E 

<D 
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in 

>< 
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Ke; 

/  Requirements  and  T&E  Measures 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

Test  Methodologies  / 
Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 


Decision 

Supported 
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Signal:  Does  the  kneeboard  laptop 

Audibility  of  emergency  signal: 

Helmet  mounted  audio 

MS  B 

emit  a  signal(s)  that  jams  or 

audible  to  subject  with  H-1 

measurement  of  emergency 

interferes  with  the  egress  signal? 

hearing  profile  (per  AFI  36-2108 

signal.  Mannequin  insert  in 

Airman  Classification)  over  90 

helmet.  Acoustic  chamber  with 

dB  noise  floor. 

simulated  noise  floor. 

Subjective  tests  in  acoustic 
chamber  using  pilots  with 
representative  hearing  (H-1  to  H- 
3  hearing  profiles). 

Kneeboard  laptop  visual  signal 

Visual  Signal  Frequency 

MS  B 

frequencies:  1.5 -2.5  Hz. 

measurement  of  kneeboard 

Q. 

O 

laptop  displays. 

Q. 

ro 

Kneeboard  laptop  visual  signal 

Subjective  determination  of 

MS  C 

T— 

CM 

~o 

k. 

ro 

o 

-Q 

Q) 

Q) 

C 

KSA: 

Personnel 

Survivability 

Can  emergency 
egress  occur  via 
ejection  seat  with 
kneeboard  laptop 

discrimination:  Differentiable 
from  cockpit  warning  signal 
frequencies  3-5  Hz  in  all  lighting 
conditions  in  frequency  and 
color. 

conflicting  and  competing 
ranges.  Blackout,  twilight,  full 
sun  and  glare  conditions. 

$ 

system  in  place? 

£ 

Automatic  kneeboard  laptop 

CASE  test  of  algorithm  using 

MS  C 

k. 

auditory  signal  cutoff:  At 

cockpit  system  and  engineering 

< 

immediate  onset  of  auditory 
egress  signal;  continuous 
thereafter. 

model  of  kneeboard. 

Kneeboard  line-of-sight  (LOS) 

Video  files  of  representative 

MS  B 

obscuration  of  cockpit  visual 

target  audience  members  in 

egress  warning  signals:  99% 

mock-up  cockpit.  Digitize  into 

visible  for  each  visibility  square 

posture  file  for  input  into  Classic 

containing  all  or  part  of  cockpit 

JACK.  Quantify  LOS  with  the 

egress  warning  display. 

JACK  Coverage  Zone  Tool 
(CZT).  Origin  location  between 
virtual  operator  eyes;  CZT 
resolution  of  100  horizontal  units 
and  100  vertical  units. 

Row  Number 

System 

Kei 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
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Critical 
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Issues  (COIs) 
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(CTP)  &  Thresholds 

UNCLASSIFIED 

92 
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CM 

CM 

Aircrew 

Kneeboard  Laptop 

KPP: 

Survivability  2 

(of  a  manned 
system) 

KSA: 

Personal  Locator 
EMI 

Can  crew 
members  survive 
vehicle 

evacuation  with 
the  kneeboard 
laptop  system? 

Personal  Locator  Device:  Does 
the  kneeboard  laptop  system  emit  a 
signal(s)  that  jams  or  interferes  with 
the  personal  locator  device  signal? 

Allowed  EMI  Interference 
between  406  MHz  emergency 
locator  signal  and  kneeboard 
laptop  system:  None. 

Test  receiver  measurements  of 
locator  signal  in  presence  of 
engineering  model  of  kneeboard 
laptop  unit. 

Frequency  start:  300  MHz 
Frequency  stop  =  1  GHz 
Resolution  Bandwidth  (RBW)  = 

120  kHz  (6  dB) 

Step  size  =  60  KHz  (1/2  RBW) 

Operational 

Test 

Readiness 

CO 

CM 

Aircrew  Kneeboard  Laptop 

KPP: 

Survivability1 

(of  a  manned 
system) 

KSA: 

Detectability 

Can  crew 
members  survive 
vehicle 

evacuation  with 
the  kneeboard 
laptop  system? 

Detectable  Signal:  Does  the 
kneeboard  laptop  emit  a  signal  or 
visually  detectable  light  source  that 
may  alert  the  enemy/foe  of 
emergency  egress/distress? 

Range  of  electromagnetic 
signals  from  kneeboard  laptop: 
Undetectable  outside  of  a 
sphere  of  two  (2)  foot  radius 
centered  on  kneeboard. 

Simultaneous  screen  automatic 
shut-off  with  initiation  of 
ejection. 

Near-field  test  receiver 
measurements  in  free-field  test 
range  of  kneeboard  laptop 
engineering  model. 

Frequency  start:  9  KHz 

Frequency  stop  =  3  GHz 

Resolution  Bandwidth  (RBW)  = 

20  kHz  (6  dB) 

Step  size  =  10  KHz  (1/2  RBW) 

Subsystem  testing  of  automatic 
shut-off  circuit  in  simulated 
ejection  scenario. 

Operational 

Test 

Readiness 

CM 

Fighter  Aircraft 

KPP: 

Sustainment 

KSA: 

Reduced 
Ownership  Cost 

Does  the  system 
create  an 
environment  that 
induces  hearing 
loss  that  creates 
a  permanent, 
debilitating 
impairment? 

Number  of  Personnel  with  time 
weighted  average  noise  exposure 
in  excess  of  85  dB  over  a  12-hour 
period.  Do  costs  of  compensating 
and  treating  airmen  for  permanent 
hearing  loss  exceed  Surgeon 

General  and  Veterans 

Administration  annual  budget 
allocations  for  aircraft  platform? 

Number  of  air  crew  and  ground 
crew  members  exposed  to  >  85 
dB  time  weighted  average 
equivalent  sound  level  over  a 
12-hour  work  shift  (annually): 

<  4 

IMPRINT  simulation  of  air  crew 
performing  missions  1,  II,  IVC  and 
VII. 

IMPRINT  simulation  of  ground 
crew  simultaneously  readying 
four  4  aircraft. 

Quarterly  audiometric  testing  of 
aircrew  beginning  at  IOC. 

PDR 

Post- 

Deployment 

Review 

2  Survivability  attributes  are  those  that  contribute  to  the  survivability  of  a  manned  system.  This  includes  attributes  such  as  speed,  maneuverability,  detectability, 
and  countermeasures  that  reduce  a  system’s  likelihood  of  being  engaged  by  hostile  fire.  These  examples  relate  to  the  detectability  or  lack  of  detectability  that 
would  put  the  system  at  risk. 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

in 

CM 

Fighter  Aircraft 

KPP: 

Sustainment 

KSA: 

Availability3 

Does  air  crew 
and  maintenance 
crew  noise 
exposure  limit 
system 
availability  for 
forecast  levels  of 
manpower? 

Noise  exposure:  Do  aggregate 
noise  exposure  levels  constrain  the 
flight  time  (range)  of  the  aircraft? 

Do  aggregate  noise  exposure  levels 
increase  the  required  manpower 
levels  for  the  ground  crew? 

Time  weighted  average  noise 
level  (with  hearing  protection): 
8-hour  time  weighted  average 
equivalent  sound  level  <  85  dB 
for  any  flight  crew  or  ground 
crew  member. 

Percent  of  ground  crew  that 
incorrectly  implement  and 
adjust  hearing  protection: 

<0.1% 

Near  Field  Measures: 

Maintenance  and  operations 
personnel  acoustics  in  dB. 

Military  power  and  afterburner 
power.  Stationary  aircraft  on  the 
ground. 

Octave  band  sound  exposure 
levels  (SEL)  in  dBA  at  500,  1000, 
2000,  3000,  4000,  5000  and 

6000  Hz  at  each  crew  position. 

Octave  band  measurements  in 
dB  at  500,  1000,  2000,  3000, 

4000,  5000  and  6000  Hz  at 
positions  on  a  six  (6)  feet  by  six 
(6)  grid  within  cone  +/- 135°  from 
aircraft  nose.  All  equipment 
operating  in  normal  mode. 

Form  and  custom  mold  ear 
protection  testing.  Twenty 
insertions,  twenty  subjects. 

Repeat  at  two  hour  intervals. 
Administered  by  AF  Hearing 
Conservation  Program  designee. 

MS  C 

3  The  Availability  KSA  for  this  example  relates  to  the  relationship  between  ground  crew  availability  and  the  ability  to  field  the  fighter  at  optempo.  Since  there  is 
a  limit  on  the  amount  of  exposure  to  individual  ground-crew  members,  a  significant  number  of  ground  crew  members  exceeding  that  limit,  would  place 
mission  capability  at  risk.  Ignoring  the  limit  would  drive  up  Air  Force  enterprise  medical  costs. 
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Row  Number 

System 

Ke' 

/  Requirements  and  T&E  Measures 

Test  Methodologies  / 

Key  Resources 
(M&S,  SIL,  MF,  ISTF.HITL, 
OAR) 

Decision 

Supported 

Key 

Requirements 
(KPPs,  KSAs) 

Critical 
Operational 
Issues  (COIs) 

Key  MOEs/MOSs 

Critical  Technical  Parameters 
(CTP)  &  Thresholds 

CO 

CM 

Fighter  Aircraft 

KSA: 

Noise 

Compatibility 

Is  the  system 
suitable  for 
CONUS  basing? 

Far  Field  Noise:  Will  relative 
geospatial  positioning  of  required 
flight  profiles,  dominant  weather 
patterns,  frequency  of  flight 
operations  and  airfield  acoustics 
combine  with  aircraft  noise  to  result 
in  noise  contours  that  are 
unacceptably  high  to  stakeholders  in 
communities  surrounding  CONUS 
air  bases? 

Sound  Exposure  Levels: 

<100  dBA,  1000  ft  flight  level , 
minimum  power 

<120  dBA,  100  ft  flight  level,  mil 
power 

<125  dBA,  100  ft  flight  level, 
with  afterburner 

Field  measures  followed  by  M&S 
using  Advanced  Acoustics  Model. 

Far  Field  Measures:  Community 
acoustics,  sound  exposure  levels 
(SEL)  in  dBA. 

Array  of  170  microphones. 

Flight  level  (above-  ground) 
measures:  0  to  300  ft 

Lateral  measures:  25  to  7  miles 

160  Knots  true  airspeed 

Advanced  Acoustics  Model 

applied  to  candidate  CONUS 
basing. 

Flight  level  (above-  ground) 
measures:  0  to  1000  ft 

Adjust  to  Standard  Acoustic  Day: 
International  Standard 

Atmosphere  +  18°F  (ISA+  18°F) 

Fielding 
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